August 2019
RESEARCH VOLUME 1:2 QUARTERLY Bringing great research ideas into open source communities
How To Train Your Model: E. Ugur Kaynar’s Research Adds Object Store Caching to Ceph, Speeds Machine Learning
Building a Linux Unikernel Has Open Source Made Patents Obsolete? Where We Came From: Red Hat Research in Brno, Czechia RESEARCH QUARTERLY
VOLUME 1:2
Table of Contents FROM THE DIRECTOR From the director 02 It is a surprising coincidence, to me at least, that the first NEWS
Red Hat Research day, part I: How do we two articles in this quarter’s RHRQ deal with the ownership reconcile privacy with machine learning? 04 of information from completely different viewpoints. In the INTERVIEW
The role of patents in the open source News section, we have an article on technology that is all Innovation cycle 07 about keeping private things private. FEATURE ARTICLES
Opening doors with open source: The ability to use private data without how we launched the University Program at leaking it is absolutely crucial to the Red Hat 11 pursuit of science and of open source Cooperative object store caching for software development, and author big data analytics 16 Gordon Haff covers a whole series Big data analytics 20 of talks on that subject. Second,
COLUMNS we have an interview with Red Hat patent attorney Jared Engstrom, who Quarterly Reflections: Living to serve 23 talks about the role patents played RIG Leader’s Perspective: How a RIG is in convincing people to make private born - building a research interest group things public—in this case, of course, in Tel Aviv 25 the private things were not personal information, but trade secrets. Engstrom talks about the patent system’s role in opening those secrets to public view, which played no small part in the unikernel, both come out of talks at our manufacturing revolution in the 19th first-ever Red Hat Research Day. We and 20th century. He also goes into brought together 150 professors, grad the position Red Hat takes on patents, students, Red Hat engineers, and and how the speed of open source Red Hat partners and customers the day About the Director: Hugh Brock is the Research Director for Red Hat, coordinating development has made them less before our annual Red Hat Summit to Red Hat research and collaboration with relevant in the software industry today. hear presentations on the innovative- universities, governments, and industry worldwide. yet-practical work Red Hat Research A Red Hatter since 2002, Hugh brings intimate Our technical articles in this issue, one knowledge of the complex relationship between supports. The event went so well that on object store caching and the other upstream projects and shippable products to the we have been challenged to do it again task of finding research to bring into the open on the development of a Linux-based source world. next year, only twice as big—so watch
02 RESEARCH QUARTERLY
VOLUME 1:2
this space for more information. Our first think about the platform business—a step will be adding a second event at our business we have been in for almost our Brno developer conference, DevConf entire history, but one IBM dominated (https://devconf.cz), in January 2020. for decades before that. What kind of research does a platform company do, You might be forgiven for thinking that and who does it benefit? I wrote an Red Hat Research began with the first essay on the subject for the Red Hat RHRQ issue, shipped just last May. In Research team, and they thought it fact, we have been at this for quite a belonged in the magazine—so here it while, mainly in our Brno, Czechia office— is. I’d love to hear your thoughts on the our largest engineering office —where subject as well ([email protected]). I was moved...to think we launched this program almost ten about the platform years ago. Brno engineer and manager, I have one other exciting thing to Radek Vokal tells the quintessentially mention: You can now subscribe to business—a business we Red Hat story of how we got started. RHRQ! Just browse to research.redhat. have been in for almost Meanwhile, in the first of a regular com/quarterly to sign up. We will mail our entire history, but series from our Research Interest Group you a print copy or email you a link, or leaders, Miki Kenneth talks about the both—your choice—and we will charge one IBM dominated for steps they took to get the Tel Aviv office you only the price of the time required decades before that. involved with universities there. to type your address. Personally, I prefer — my magazines in print… showing my age, Last, but I hope not least, I was moved I know. What kind of research by the recent, long-awaited closing does a platform of the IBM-Red Hat acquisition to company do, and who does it benefit? Red Hat Research Quarterly delivered to your digital or physical mailbox?
Yes! Subscribe at research.redhat.com/quarterly.
03 RESEARCH QUARTERLY
VOLUME 1:2
HOW DO WE RECONCILE PRIVACY WITH MACHINE LEARNING? Two critical areas experiencing a wave of innovation and disruption were the focus of the Red Hat Inaugural Research Day in May. The first was data sovereignty in all its aspects (privacy, governance, security) and the second was hardware and microarchitecture innovations for complex, data-intensive, distributed computing systems. There’s a clear linkage between these two areas (the former often requires performance provided by the latter), however, they involve distinct communities of experts that don’t often interact. One of the objectives of Red Hat Research has been to encourage collaboration across such communities.
This article is focused on the track that covered reconciling privacy with machine “Data is useful. It may learning. These developing technologies are especially pertinent when applying AI even be the new oil... techniques to problems where lots of data exists but sharing and pooling that data creates privacy concerns, think medical imaging, for example. In a future installment But data can also of RHRQ, we plan to dig into the track that covered the complementary hardware and embed information that operating system work needed to accelerating those and other techniques. identifies individuals PRIVACY AND MACHINE LEARNING: even if it has been anonymized in some and their personal AN INHERENT CONTRADICTION? manner.
characteristics—which Data is useful. It may even be the new Alternatively, a company may consider a they consider deeply oil. Healthcare can use large data data set a trade secret and may not want sets to correlate different treatment private.” to share it for that reason. That said, approaches with health outcomes. But anonymization and related techniques data can also embed information that are widely used in healthcare and identifies individuals and their personal other industries and this is an area of characteristics—such as some aspect of considerable study on its own. (As we’ll their health—which they consider deeply see when we get to differential privacy) private. However, if our primary use of a set Anonymizing the data is one obvious of data is to train a model, we have approach. But effective anonymization additional options to obscure data. is surprisingly difficult, given enough Sharing only the model derived from the data points and the ability to combine training data, rather than the data itself, additional data from a variety of sources. helps to some degree. But research has Furthermore, regulations may place demonstrated ways to tease out hidden restrictions on sharing healthcare data
04 RESEARCH QUARTERLY
VOLUME 1:2
data from models derived from it so practical lately because of algorithm decrypt any of the inputs but all have just sharing the model by itself is not a and computing improvements. There’s access to the data outputs (such as with complete answer. also just a lot more interest in figuring the average salary by gender in the City out ways to share outputs (such as of Boston example mentioned above). Three specific approaches were models) for a data set without either covered in the course of the Research HOMOMORPHIC ENCRYPTION sharing the data itself or involving a Day morning track chaired by Azer trusted third-party. While technically distinct from MPC, Bestavros of the Hariri Institute for homomorphic encryption can be used Computing at Boston University Essentially, MPC replaces a trusted to tackle a similar class of problems. The (BU): secure multi-party computation third-party with a protocol. (In this sense, strongest variation on homomorphic (MPC), homomorphic encryption, and it has some conceptual similarities with encryption is fully homomorphic differential privacy. blockchain.) This includes preserving encryption (FHE), which allows for certain security properties, such as SECURE MULTI-PARTY arbitrary functions to execute on COMPUTATION encrypted data. Homomorphic Machine Learning: Actionable Knowledge from Data Encryption is essentially a Imagine that a number of technique to extend public-key companies have data that cryptography and was, in fact, they are willing to allow TRADE first mentioned shortly after the government or other Persisted SECRET model the RSA cryptosystem was organization to use for initially invented. some purpose—but only so long as no one else can see At Research Day, Kurt Rohloff, their particular data. This CTO and Co-Founder Input Prediction was the case with a project g(x ,x ,x , ...) Results of Duality Technologies, PRIVATE data 1 2 3 done by Boston University discussed how homomorphic with the City of Boston encryption could, for example, regarding gender-based allow an owner of sensitive wage gaps. privacy and correctness, even if some genetics data to encrypt it, send it of the parties collude and maliciously to some external provider or other Historically, a requirement to keep data attack the protocol. MPC is an area organization for analysis, and then secret—even at an aggregated level— of active research and there are many receive back an encrypted result—that would have been a showstopper. Can variants, especially once one gets only they can decrypt. In other words, you share data without actually sharing beyond “toy” problems. In general, you homomorphic encryption lets a third it? That’s where MPC comes in. can think of MPC distributing shares party perform complicated processing of The science behind MPC dates back of secrets among the parties doing data without being able to see it. about 30 years but it’s become more the computation. No single party can “Ordinary” encryption schemes make it
05 RESEARCH QUARTERLY
VOLUME 1:2
virtually impossible for someone without However, in return, you get results that a secret decryption key—such as a cloud can’t be reverse engineered to derive provider—to manipulate encrypted data any specific input. in a useful way. However, homomorphic CAN WE CHANGE THE or malleable encryption schemes allow TRADEOFFS? you to do so. Ensuring such schemes preserve security attributes and allow Even attributing good intentions complex operations is part of ongoing to data users—certainly not always research, as is improving efficiency, warranted—there are tradeoffs between since current FTE is too computationally extracting value (from the perspective intensive to be generally useful. of both companies and consumers) and maintaining privacy. Perhaps it’s not DIFFERENTIAL PRIVACY always either/or. But that’s mostly the Suppose you want to widely share default assumption. “Perhaps we don’t need statistics over a set of data but don’t However, we’re seeing a growing toolbag want to or can’t share the individual data to choose data over of techniques, even if many of them are points because of privacy concerns? privacy if we can apply still squarely in the research phase, that This need for anonymization, touched seek to make other tradeoffs instead. more sophisticated on earlier, is hardly new. After all, it’s Perhaps we don’t need to choose data algorithms to the a concern with just about any survey over privacy or privacy over data if we problem.” that publishes its results. And it’s been can, instead, apply faster computers or “solved” in many different ways—often more sophisticated algorithms to the well enough but vulnerable to known problem. attacks and offering privacy protections Visit the Red Hat Research Program. that are not well-defined.
In a nutshell, a differential privacy AUTHORS algorithm injects random data into a — Gordon Haff, Technology evangelist, data set (in a mathematically rigorous Red Hat way) to protect individual privacy. — Kinan Bab of Boston University Because the data is “fuzzed”, to the covered multi-party computing extent that any given response could instead have plausibly been any other Article links: It may even be the new oil – https://www.economist. valid response, the results may not com/leaders/2017/05/06/the-worlds-most- be quite as accurate as the raw data valuable-resource-is-no-longer-oil-but-data depending upon the technique used. Red Hat Research Program – https://research. redhat.com/
06 RESEARCH QUARTERLY
VOLUME 1:2
OPEN SOURCE AND PATENTS: RECOGNIZING INNOVATION JARED ENGSTROM WITHOUT BUILDING FENCES Manager, IP Attorney at Red Hat Hugh Brock, Director of Research at Red Hat, had the opportunity to interview Jared Jared Engstrom is an intellectual Engstrom from the Red Hat legal team. As an IP attorney, Jared manages the Red Hat property (IP) attorney with a background patent portfolio and has become a specialist in open source patent law. in electrical and computer engineering. Since joining Red Hat in 2012, Jared has RHRQ: Can you tell me how you came I think everyone agrees that the patent managed Red Hat’s patent program, to open source and intellectual property system was originally designed to including work on IP issues at the (IP) law and what you continue to find promote innovation and to share ideas. intersection of patents and open source interesting about it? You publish your ideas and you share software. those ideas with the world, and in Jared Engstrom: Well, honestly, I exchange for that, we’re going to give lucked into it. I started my career at a you something. In the case of the United law firm writing patent applications and States, we’re going to give you a 20-year then worked at Hewlett-Packard (HP) monopoly for that idea. There are people for several years, a company with a rich today who suggest that the patent history of patents and innovation. In system is not doing that as effectively 2012, a recruiter from Red Hat reached as when it was originally designed, and out to me with a job offer. They were that the focus for many patent holders looking for someone to manage the Red is more about excluding the use of new Hat patent portfolio, and it immediately technologies rather than sharing them. intrigued me. The world of open source One of the things I’ve discovered is that wasn’t something that I was fully open source, in some ways, has become enmeshed in. I had dealt with some open the more effective way to promote source issues at HP, but open source innovation. A lot of the truly interesting, there was largely a compliance effort, meaningful ideas, certainly in the making sure they were managing all their software realm, are coming out of open software in compliance with relevant source. And, even though Red Hat may open source licenses. The Red Hat offer be one of the few “purist” open source was extremely interesting because the companies, all of the big tech companies relationship between patents and open that are having success, in some way source is not obvious to most people or another, are using open source as a and, when I tell people I’m the patent means of doing that. guy at an open source company, I get lots of questions: How can that be? How RHRQ: It had not occurred to me that does that work? open source had really picked up that banner or promoted innovation from
07 RESEARCH QUARTERLY
VOLUME 1:2
the patent system. It’s a very get your name on a patent innovation but, because interesting way of looking at as an inventor, that’s good they don’t usually productize it. for your own reputation and what they’re doing, the it’s good for the university’s options for getting a return JE: Not everyone is going reputation. That’s because on that investment are to agree with that, but as patents are held out as a limited without monetizing somebody who now drinks symbol of innovation, and the IP—particularly patents liberally from the open source you can quantify that and —associated with that fountain, I think the idea has measure it. One of the areas research. A lot of patent real merit. where I think the patent trolls are getting their RHRQ: In working with system still shines, is that it patents from other operating universities, how general do is the most comprehensive, companies. Historically they you find that understanding easily accessible, easily have gotten some from to be? I know, from talking searchable, readily available universities as well. Some to some of my university way to quantify how much companies are less likely to colleagues, that the old innovation is going on in an want to do joint research university tech transfer and organization. Whether it’s with universities if they know patent licensing offices are at a university, a company, that those universities are no longer seen as a great or wherever, everybody then turning around and way of raising revenue for has patents, and you can trying to monetize those the university. What do you measure that and have patents via patent trolls. It’s find the attitude to be there that stand as a proxy for a difficult position to put a in terms of patenting things the level of innovation. university in, and, at least at a versus publishing them With the rise of the patent high level, the approach that allowing them to be free? trolls and the disdain that Red Hat appears to be taking a lot of companies have represents, I think, a different JE: I don’t have deep towards them, the idea way that may have some connections with folks in the of purely monetizing your long-term advantages for the universities to really know intellectual property has universities. where their mindset is on become disfavored in certain RHRQ: You touched on that, but to the extent that technology environments, something that resonated I have talked to individuals, such as software. Universities with me about the I get the impression that that do a lot of research in reputational benefits of being there is more interest, the software space are left in on a patent. Certainly being priority, and concern over a difficult position, because first with an idea is the coin the reputational benefits of they are doing great research for even getting tenure—it’s doing patent work. If you can and adding real value to not merely reputation, it’s 08 RESEARCH QUARTERLY
VOLUME 1:2
how do you succeed at all as of publications and patents. patents. In fact, it is good in an academic? You publish some ways that we own this On the patent side, how do papers or you get your name intellectual property. But you provide that reputational on patents or “ideally” both. we’re going to make them benefit while simultaneously The question is, what can we open to the community, avoiding some of the do in open source to provide and here’s all the things perceived downsides of that kind of prestige: to credit that we’re going to do to patents? The way we’ve the invention of a thing, ensure that.” We have the tried to approach it at Red without encumbering it with Open Invention Network Hat is to say, look, there is a bunch of obligations? (OIN), which is the world’s nothing inherently bad about largest patent cross-licensing JE: I love that question, patents. To me, I often use organization. Anybody who because I have lots of the analogy of real estate joins OIN gets a patent thoughts on that, even if transactions, real property. If license to any patents that some of them are not fully you’re buying a house, you’re relate to Linux. We also have baked.I am not a developer buying a piece of property. the Red Hat Patent Promise, per se, although I do have There is a boundary line which is a further assurance a background in electrical around that. That piece of that Red Hat is not going to engineering and took property is yours, you can enforce its patents against coding classes in college buy it and own it, and what anyone who makes, uses, or and so forth. But I’m on you do with it really depends sells open source software. Github and I’m familiar on what your priorities Additionally, we helped form with the mechanics of how are. You can put a big the LOT Network. LOT is an repositories work. There are fence around that piece of anti-patent-troll network ways of measuring one’s property, and prevent anyone where anyone in the network level of activity on some from going onto it or extract agrees that if you transfer of these repositories. I get fees from anyone who wants one of your patents to a the sense that there are to go onto the property. Or, patent troll, everybody else numbers-based reputational you can turn it into a public in the network gets a license benefits to that. There is sort park, and say “I own it, but to that patent. So, if you put of the social component of because I own it, I am going all these protections around just getting to know other to preserve it as an open patents, then you can still developers and developing space that anyone can use.” have the public recognition relationships and trust and In general, the approach and reputational benefits that sort of thing. But I that Red Hat has taken with of documenting before don’t get the sense that the patents is to say “there’s a government that you reputational benefits have nothing wrong with owning have invented something yet risen to the universal level 09 RESEARCH QUARTERLY
VOLUME 1:2
innovative and useful, and here is the terms of how it might be handled. RHRQ: Is there any reason that a inventor’s name on it. If you put the One approach we have considered university should be afraid to work with appropriate protections around it, is to have both parties agree “in us in an open source context? What do there shouldn’t be any downsides to writing” that the priority is to make they have to fear? that, either for the inventor or for the the research outcomes open and to company that’s doing it. That’s the Red make any technology developed under JE: Well, I guess maybe I would flip Hat approach in a nutshell. the agreement available through an that on its head a little bit, and say what open source license. Specifically, with are the advantages of working with us The thing that I go back to again and respect to patents, any patentable ideas in an open source context? Given the If you put protections developed solely by Red Hat would rate and pace of technological change around patents, then you be owned by Red Hat, any patentable today, especially in software, there ideas developed by the university would are real advantages to working with a can still have the public be owned by the university, and any company that is adept at adapting to recognition and reputational patentable ideas that are developed change. I think one of the benefits of benefits of documenting together, would be jointly owned. getting involved with Red Hat is that we have developed the skills of a surfer. I before a government RHRQ: Let me touch on a little different think Red Hat has become pretty good that you have invented area. We produce a lot of software in at identifying those technological waves these university partnerships. Do we something innovative and as they come in, catching them and have opinions or a position on the open riding them, and then getting off when useful... source licenses that we should use? Is the wave is losing energy and preparing there a preferable one and why? for the next one. Innovating becomes again is the analogy of the real estate, less about building static fences around JE: My understanding is that in true the real property concept. By owning ideas and more about being able to ride open source fashion, we generally that piece of property, you get to waves. No wave lasts forever, but if you leave it to the individual developers to control that piece of property’s destiny. can dynamically leverage the energy make the choice about which license You can either make it closed or you from one wave into the next, then you makes the most sense. Obviously, there can make it open. The more patents we create increasing value over time. So I are licenses that offer more benefits in can get, the more we can make those think learning to ride and cycle through terms of how they apply to patents— spaces open for collaboration within the those waves efficiently and effectively certain open source licenses have community. is a huge benefit to a university. either implied or explicit patent license RHRQ: In your experience doing our grants, the General Public License agreements with universities to date, do (GPL) being a primary example of we have a position on patents that arise that. However, many developers today Article links: out of that joint work that we do? prefer the more permissive open Open Invention Network – https://www. source licenses such as Apache or openinventionnetwork.com/ JE: This is still an emerging area in Business Source License (BSD). Red Hat Patent Promise – https://www.redhat.com/ en/about/patent-promise 10 LOT Network – https://lotnet.com RESEARCH QUARTERLY
VOLUME 1:2
OPENING DOORS WITH OPEN SOURCE: HOW WE LAUNCHED THE UNIVERSITY PROGRAM AT RED HAT Open Source was not always the default choice for software development. Research groups have not always seen the value in opening up not only the code, but also their research data, processes, plans and communications.
They say that change is hard and that themselves. Open source was not then universities have a long history and set the default choice for development, and of experiences that make introducing most of the faculty still didn’t believe radical changes “in the way they think that the open model for development and work” even harder. So why did and collaboration could work. This has it change, and how have universities been especially true for the research started benefiting from the open model community who are very protective of as seen from the work Red Hat has done their algorithms and data which they with the Czech Republic? think of as a “competitive advantage” over other researchers. Therefore, after Red Hat started operating in the Czech forming the office and after two years Republic in mid-2004. One of the main of growth, in 2007 Red Hat went to the reasons to open an office in this country universities to convince them otherwise. was the significant participation of Czechs in various open source projects It wasn’t easy at first—there were a such as the Linux® kernel, Gnome and couple roadblocks that we had to KDE desktops, and low-level tools and overcome. The first roadblock was that compilers, such as Apache and GNU open source was not perceived as a Compiler Collection (GCC). Back in valuable and sustainable development those days, all of these folks were either model. Additionally, the universities students or post-docs at one of the were strictly opposed to publishing the universities who contributed to open code of the projects their faculty and source in their spare time or as part of students worked on, even when it was their research projects. an undergraduate thesis or simply a regular small project for their classes. Red Hat hired them, grouped them The fear of having someone copy the together, and started an office that work of someone else was perceived as has become a very successful software bigger risk than the potential benefits engineering hub. However, with this of being open. approach we realized that we were leaving a gap at the universities Another interesting roadblock that
11 RESEARCH QUARTERLY
VOLUME 1:2
we hit—one which we had already heard about worked and they questioned hadn’t even thought about open source and were the participation of the before—was money. All of the open to helping us out. We student. They asked, “How universities were set up in started bringing interesting can they prove this is their a way that required anyone people into discussions with work and their ideas?” “Why from the outside wanting students, usually through the is he showing us code that to do anything with their professors we worked with. someone else had to fix after students had to pay: per They would let us “hijack” him?” “This seems like it is student, per year, or via some an hour or two of their class someone else’s work, not his and talk about open or hers?” We still had a long source and why we way to go, but the students think it’s the future. helped us a lot.
We approached Thanks to several professors, some students with especially Vaclav Matyas open source-related from Masaryk University projects, typically and Tomas Vojnar from Brno in the form of a University of Technology, we master’s or bachelor’s kicked off some joint efforts. thesis, that required The whole idea was to hold the student to work meetings and bring new closely with an ideas and experiences to the upstream community students. What we started and contribute back doing back then we would to the project. Of probably call meetups today, course, because the projects but this was not common Radek Vokal (center) answers fee that always came up in were all open source, the 10+ years ago. We organized questions from Brno developers discussions. Our budget was and students after a talk at Devconf resulting work was something courses and workshops not ready for that. 2009. The event, small at the start, that had to be shared based where students would come has grown to 1600 participants last year, the max capacity for the Brno START WITH THE on the license the upstream voluntarily without getting University of Technology venue where STUDENTS project had. I remember any academic credit. We it is held. So, we started with that the first project we started with what we knew evangelism. We talked to did this way caused a huge best: Linux administration, students we already knew discussion amongst the including going into the and professors whom we professors judging the work. details; hacking in C or knew from our student It seemed 4 out of 5 of them Python; and even doing days, especially those who had no idea how open source simple things like holding
12 RESEARCH QUARTERLY
VOLUME 1:2
install-fests. Remember, at DevConf US (Boston, August the connections they made this time, even installing a 15-17, 2019). through us. We introduced Linux distro involved a lot them to big names in the We built strong connections. of manual tweaking. We field and we invited visible Most of the folks we hired talked about open source community contributors through these university principles and licences. The to speak with them. This activities wanted to students loved it! We were made a huge impact on the contribute back. What getting more students in universities. All of a sudden, started as a small effort these free course than some we had several open source by a few people grew professors were getting in exponentially in their traditional courses. several directions. Based on these courses, All of a sudden, we we started getting lots of had courses and labs interest from students in how focused on operating they could work more with us. systems-level topics They wanted to do projects and, thanks to or internships at Red Hat. people from JBoss, It was an amazing time. At a new focus on the one point we had so much application layer. content that we wanted to Keep in mind, we still share at the universities that had zero funding we started organizing our and this all grew very own conference, DevConf. organically. cz. The conference started At this point, the at Masaryk University in university representatives projects and research groups 2007 and had approximately Stunned surprise: Some talks at realized that there could approaching us and asking Devconf really catch people off guard. 20 sessions of talks and be benefits for both sides. us to help them with open workshops. Back then this After a couple years of source and how they could was a huge number! In us approaching students collaborate, not only with 2018, we had over 4,000 directly, the universities us, but how they could build attendees, almost 280 realized that this is what a community around their speakers, and participants the students wanted. The projects. from all over Europe. The students desired a close event has since spawned TURNING PROJECTS cooperation with experienced DevConf IN (Bengaluru, INTO PRODUCTS people from an international August 2-3, 2019) and company. Students valued Our first thought had always
13 RESEARCH QUARTERLY
VOLUME 1:2
been to focus on the students and A research project started at Brno researchers. As long as there was an idea University of Technology works on how to build something in an open source to collect and analyze these thousands way and someone had spare cycles to of reports and find similarities in the work with that project, we would help. core dumps. That required a way to get We never thought about these research the most valuable information into a projects as something Red Hat would bug report so that a package maintainer immediately benefit from. As it turns could fix the issue. The project is called out, there was such a wide range of FAF, Fedora Analysis Framework, and projects that it has been used successfully in the many of them Fedora infrastructure for the last couple have proven of years. It evolved into a server which to provide collects tens of thousands of ABRT very tangible crash reports per day and provides benefits. fast detection of duplicate problems, statistics, and clustering functionality. These include projects such as Other projects followed a similar path. ABRT, Automatic We had people using different tools Bug Reporting for static code analysis. Most of these Tool. Developers tools were proprietary and they had who use Fedora never really been run against large code might know bases such as the Linux kernel. A few this tool well. engineers from Red Hat started working It collects all closely with a research group called Ten years later, the AV is somewhat less of a crashes on your system and reports Verifit and improved the performance challenge, but Devconf still requires hours of them back to the project maintainers. and the results of their tools based on volunteer time from Red Hat IT. The idea for this project had been analyzing the very large code bases that around for some time, but we never are present in the open source world. found the time and resources to kick it Some research topics were also found by off. We had a simple proof-of-concept just talking to developers and analyzing that quickly uncovered that we needed the patterns of how they use things. something more scalable. If a kernel There was one done by a student from crash appears on ten thousand boxes, Masaryk University which looked at the you don’t want each one of them to potential security issues that can happen open a bug report including all the same when a developer misunderstands the logs and data.
14 RESEARCH QUARTERLY
VOLUME 1:2
documentation of an API. As a result approach mutually reinforce each other. of this project, the student submitted And there are new projects starting all patches to the OpenSSL project and the time. You can see the list of active documented the most common issues projects at https://research.redhat.com. that he found. Feel free to watch their progress, or even contribute, since the work is being Last, but not least, a project in a conducted upstream in the open source completely different category combined projects themselves. We are always the skills of several different groups. interested in receiving new research The project is called Robomise.cz and at proposals that can be supported by our first glance it looks similar to the various engineers in the Czech Republic, and Scratch implementations designed we’re delighted that we’ve been able to teach coding. The project brought to take the approach we developed in together work on an infrastructure to Brno and extend it to other universities run on, a user-experience designer to around the globe. design a nice interface for it. On top of How have universities that there were a few people looking at AUTHOR started benefiting from adaptive learning and how to change — Radek Vokal, Senior engineering the open model, as the task level based on progress on manager, Red Hat previous tasks. This involves monitoring seen from the work Red the activities performed during the tasks Hat has done with the using machine learning to understand Czech Republic? how many times a person is making Article links: errors or succeeds. This information is Vaclav Matyas – https://www.muni.cz/lide/344- vaclav-matyas then integrated into the UI to advance Tomas Vojnar - https://www.fit.vutbr.cz/~vojnar/ the user by several levels or roll them index.php.en back to a previous one if they need to DevConf IN - https://devconf.info/in relearn foundational skills. As you see, DevConf US - https://devconf.info/us this is not a project that Red Hat would FAF - http://retrace.fedoraproject.org/faf ever plan to productize and support, but, Verifit - http://www.fit.vutbr.cz/research/groups/ thanks to this work, we have students verifit/.cs who successfully develop on Red Hat OpenSSL - https://www.openssl.org/ OpenShift and others who use it as an documented - https://research.redhat.com/ AI/ML platform. academic_pubs/why-johnny-the-developer-cant- work-with-public-key-certificates/ There are many more examples of how Robomise.cz - http://robomise.cz academic inquiry and the open source https://research.redhat.com
15 RESEARCH QUARTERLY
VOLUME 1:2
COOPERATIVE OBJECT STORE CACHING FOR BIG DATA ANALYTICS The Datacenter-Data-Delivery Network (D3N ) is a novel multi-layer cooperative caching architecture for object stores. The D3N is designed to accelerate big data analytic workloads with strong locality traits and limited network connectivity between compute clusters and data storage. It started as an academic research project, but now the code is “upstreamed” as a Ceph open source project (RADOS Gateway) and will be deployed in the Mass Open Cloud production cluster.
The ability of an organization to make repositories, called data lakes, to store good use of data analytics depends on and share terabyte and petabyte- the amount and variety of data it can scale datasets. By necessity in large collect, and the speed at which it can organizations, distributed big-data glean critical insights from that data. analytic clusters such as Hadoop and Interestingly, these two factors are Spark must depend on accessing somewhat in conflict—the more data a centrally located data lake that is you collect, the longer it takes to do the relatively far away. Even with a well- data-intensive model training work that designed datacenter network, cluster- is required to gain the insights you need. to-data lake bandwidth is typically The caching work we describe below is much less than the bandwidth to focused on reducing the problems this storage within the compute clusters. conflict causes. Consequently, users will manually copy a repeatedly accessed dataset to local storage (e.g. HDFS), incurring complexity and performance overhead to manage data placement and replication and to maintain consistency between replicas.
To address these network limitations we designed a multi-layer throughput- oriented caching architecture, Datacenter-Data-Delivery Network (D3N), that uses high-speed storage
The more data you collect, the longer it takes to To help customers do large-scale such as NVMe flash or DRAM to cache do the data-intensive model training work that is analytics today, many datacenters datasets on the access side of each required to gain the insights you need. include low-cost, centralized storage layer of a hierarchical network topology.
16 RESEARCH QUARTERLY
VOLUME 1:2
Such caching allows big data jobs to use organized into pools of different sizes, the capacity available at each network with cached data, in all but Level 1, layer in an oversubscribed network. distributed across these pools via D3N dynamically allocates cache space consistent hashing. The resulting logical to different caching layers to maximize caches form a traditional caching utilization of each network layer. hierarchy, where caches nearer the client have the lowest access latency and We implemented D3N as modifications overhead, while caches in higher levels to Ceph’s RADOS Gateway (RGW), in the hierarchy are slower (requiring a Ceph component that supports the multiple hops to access), potentially S3/Swift-compatible object interface larger (incorporating storage from supported by most big data frameworks. more individual cache servers), and D3N requires no changes to the shared by more clients. The L1 cache interfaces of any Ceph services, involves server nearest to the client handles no additional metadata services (e.g, to object requests by breaking them into D3N improves the locate cached blocks), and implements blocks, returning any blocks which are performance of big- all policies based purely on local cached locally, and forwarding missed information. data jobs running in requests to the block home location (as analysis clusters by HOW D3N WORKS determined by consistent hashing) in the next layer. Cache misses are forwarded D3N improves the performance of big- speeding up reads and to successive logical caching layers until data jobs running in analysis clusters by writes to the data a miss at the top layer is resolved by a speeding up reads and writes to the data request to the data lake. lake... lake. The D3N architecture as shown in the figure above has three components: D3N supports:
• cache servers, to which client requests • Read cache where local requests are are directed and act as proxies for the stored in L1 and global requests are back-end object store, storing data stored in L2. locally for reuse • Dynamic cache partitioning algorithm • lookup service, used by clients to for partitioning the cache space into locate their “local” cache layers to minimize mean request • heartbeat service to track the set of latency. The algorithm uses the active caches observed throughput and access patterns to optimize the division of Cache servers are located in the cache capacity between rack-local datacenter on the access side of and cluster-wide data. potential network bottlenecks, and
17 RESEARCH QUARTERLY
VOLUME 1:2
• Write-back and Write-through cache a large data repository for the research for applications requiring different community. Between May to July 2019, reliability guarantees; these modes Dataverse provides with 407,000 files can be selected on a per-object basis. and 8.5 million file downloads. The real time statistics about Dataverse can be • Read-Ahead and User-directed found here. prefetching mechanism to improve the performance of analytic jobs even D3N can speed up the download time further. for these datasets. In addition, the PERFORMANCE D3N cache will provide a key benefit for enterprises like MOC core partner D3N is highly performant and can Two Sigma (https://www.twosigma. support per-cache read speeds of 5 com/), which uses the public cloud to GB/s, fully exploiting the SSDs and NICs run its data analytics cluster. With D3N, in our system. We also demonstrated Two Sigma can store their frequently with large scale datacenter traces that accessed datasets near where they are The project is the D3N achieves significant performance being used for compute, and decrease improvements for realistic workloads— result of two years of the requests to their private datacenter up to a 3x reduction in runtime vs. back end. collaboration with two uncached when bandwidth-constrained. universities, faculty, WHERE NEXT WHO WILL USE D3N students and software Last June, the MOC storage team We are currently working on a plan with presented our future storage vision engineers. Intel to deploy D3N cache infrastructure and ongoing efforts around D3N to with high-speed Optane storage in the the Red Hat Ceph Storage team, as Massachusetts Open Cloud (MOC) well as Ceph contributors from Intel. cluster, backed by the 20PB Northeast During the meeting, we brainstormed a Storage Exchange storage cluster. broad research agenda that will shape We expect the D3N cache to improve the priorities of the project moving big data application performance in forward, maybe even launching new the MOC, delivering great value to collaborations. researchers like me who benefit from data analytics (ML) to drive their The next step for the D3N project is to research. support a proper persistence layer in the Ceph Rados Gateway that could be used Another good use case for D3N is the for a write-back cache. Additional areas Cloud Dataverse project, which provides for follow-up work include the following:
18 RESEARCH QUARTERLY
VOLUME 1:2
• DAG based prefetching integration of Mohammad Hossein Hajkazemi EMINE UGUR KAYNAR is a Ph.D. write caching (Northeastern University) student in the Department of Computer • Directory based caching and caching Ata Turk (State Street) Science at Boston University working with Prof. Orran Krieger in the for the WAN in cloud environments Raja Sambasivan (Boston University) Systems Research Group. Her research • Investigating cache placement Amin Mosayyebzadeh (Boston interests lie broadly in the fields of problem to determine what, where and University) storage systems, cloud computing, and when to cache objects. Larry Rudolph (Two Sigma) data-intensive computing. • Implementing a metadata cache Peter Desnoyers (Northeastern The D3N project is a project of the Red University) Hat Collaboratory at Boston University. Orran Krieger (Boston University) It started as a research project by Boston University and Northeastern Matt Benjamin (Red Hat) University students. With the help of Brett Niver (Red Hat) Red Hat engineers who mentored the Ali Maredia (Red Hat) research and provided input into the design and implementation of D3N, the Mark Kogan (Red Hat) modification code was transferred to the AUTHOR Ceph open source community. It is now available on GitHub where anyone can — E. Ugur Kaynar, Boston University access it to examine or try.
To learn more about D3N project and its architectural details, visit https://massopen.cloud/d3n/ Article links: RADOS Gateway – https://ceph.com/ceph- storage/object-storage/ You can find the GitHub repository here: Mass Open Cloud - https://massopen.cloud https://github.com/maniaabdi/ceph.git Massachusetts Open Cloud - https://massopen. The project is the result of two years cloud of collaboration with two universities, Northeast Storage Exchange - https://www. faculty, students and software mghpcc.org/big-data-monster-storage-the- engineers. northeast-storage-exchange/ Cloud Dataverse - https://dataverse.org/ Credit goes to: presentations/cloud-dataverse-0 The real-time statistics about Dataverse can be E. Ugur Kaynar (Boston University) found here - https://dataverse.org/metrics Mania Abdi (Northeastern University) GitHub - https://github.com/maniaabdi/ceph.git
19 RESEARCH QUARTERLY
VOLUME 1:2
DELIVERING OPERATING SYSTEM SERVICES WITHOUT AN OPERATING SYSTEM For operating system engineers, a couple of questions consistently loom large: How much processing speed can an OS facilitate? How much of operating-system-like services are actually needed in a given use case?
There is of course much more to unikernel prototype that demonstrates operating-system science besides speed how some simple changes can bring and size, but these two dimensions dramatic performance advantages to can limit or expand the scope of an Linux unikernel apps. These changes application’s viability. The size and speed also reduce the footprint and resource race is currently playing out in the area requirements of the unikernel, to enable Unikernels have of the unikernel. running it on a breadth of devices such as those proliferating in edge demonstrated some Unikernels have demonstrated some deployments. One of the promising advantages over traditional operating advantages over use cases is for cloud providers who systems by offering an alternative deploy client workloads in dedicated traditional operating method for delivering operating system environments, benefiting greatly from systems by offering an services without requiring a separate the small memory footprints and short operating system. The other advantage alternative method for boot times of unikernels. Unikernels that makes them attractive is their deployed within a microVM have shown delivering operating customizability. In fact, they are so six- to ten-fold improvement in boot system services without popular in many important domains, times over containers. Similarly, a that some propose that they represent requiring a separate micropython unikernel had an image a significant challenge to Linux’s size of 1MB and required only 8MB of operating system. dominance. memory to run. On the contrary, we believe that BALANCING SIZE AND SPEED unikernels’ advantages represent the next natural evolution for Linux, as The approach we took with our research it can adopt the best ideas from the was to turn Linux itself into a unikernel unikernel approach and, along with by adding support within the codebase Linux’s battle-tested codebase and to build a target application into an large open source community, continue optimized unikernel binary, while to dominate. A group of researchers bypassing kernel features that are from Boston University and engineers unnecessary for running the application’s at Red Hat have created an early Linux workload.
20 RESEARCH QUARTERLY
VOLUME 1:2
Figure 1 compares the performance of where the kernel is statically linked to a TCP echo server, written in C, running run a single application. Only with this as a userspace process on Linux with approach can we enable configure the same TCP echo server linked into time and link time optimizations that a unikernel. In both cases the server is are not possible if arbitrary user- level deployed as a VM on QEMU/KVM on a applications can be run alongside the host machine and the client is running on application we are optimizing for. that host machine. While this is obviously Here’s an overview of the process we a “toy” example, it’s encouraging that followed for creating the unikernel (UKL) the unikernel version of the application prototype which uses Linux (v5.0.5) and achieves an average latency half of that glibc (v2.28). of the userspace application and a tail latency that is 41% faster.
Protection and isolation of applications • We added a new kernel configuration Figure 1. provide even more motivation for option to specify compiling the Linux unikernel research, because an kernel as a unikernel. application equipped with a library OS • We created a call to an undefined can be made to run in highly-restricted symbol (protected by an #ifdef) execution domains, such as an SGX that can be used to invoke application enclave or behind a set of software- code rather than creating the first defined interface. userspace process. IMPLEMENTING THE PROTOTYPE • We built a small UKL library which We chose a pure unikernel approach has stubs for syscalls to hide the
21 RESEARCH QUARTERLY
VOLUME 1:2
ALI RAZA is a PhD student in the details of invoking the required kernel the information security aspect of the Department of Computer Science functionality now that the syscall full stack—if there is only one process, at Boston University, working with instruction is no longer used. there is no need for privacy and a single Prof. Orran Krieger. He is interested • We changed glibc so that instead ring buffer is sufficient, enabling true in Operating Systems research, and is of making syscalls into the kernel, it zero-copy networking. Once these working on the Linux Unikernel project. makes function calls into the UKL optimizations are accepted as part of the He has been working with Red Hat since library. Linux kernel code, unikernels would be the summer of 2018. available for adoption in many use cases, • We changed the kernel linker script to from cloud to edge and throughout define new segments such as thread enterprise IT. local storage (TLS) segments which are present in application ELF binaries. We invite you to watch our progress, or • We added a small amount of even better, help us improve and further initialization code before invoking the optimize the unikernel prototype. Our application to replace initialization project is available on github: https:// normally done by user level code, e.g., github.com/razaaliraza/ukl for network interface initialization. To learn more, you can find a • We modified the kernel linking stage comprehensive whitepaper here: https:// PARUL SOHAL is a PhD student in to include the application code, glibc, razaaliraza.github.io/papers/UKL.pdf the Department of Computer Science and the UKL library to create a single We would like to thank the core industry at Boston University, working with Prof. binary. Orran Krieger and Prof. Renato Mancuso. partners of Mass Open Cloud (MOC) i.e., WHERE WILL THIS PROTOTYPE Red Hat, Intel, Two Sigma, NetApp and Her interests lie in operating LEAD US? Cisco for supporting this work. Partial systems, hardware and performance If the unikernel model becomes an support for this work was provided by the optimizations. She has been working with accepted part of Linux, our goal USAF Cloud Analysis Model Prototype Red Hat since summer 2018. would be to enable the same kind project, National Science Foundation of rich optimization that has been awards CNS-1414119, ACI-1440788 and achieved in existing research unikernels. OAC-1740218. For example, one could expose an AUTHORS alternative interface to read which does not dictate the location of the buffer — Ali Raza and Parul Sohal, but lets the kernel dictate the location Boston University (e.g., in a ring buffer or DMA buffer). Another example of an optimization is exposing a flattened version of the network stack which does not implement
22 RESEARCH QUARTERLY
VOLUME 1:2
LIVING TO SERVE: WHY RESEARCH ON A CLOUD PLATFORM MATTERS A few weeks ago I found myself sitting in San Jose writing a note to the Red Hat Research team about the importance of the software we build at Red Hat, looked at through the lens of our upcoming combination with IBM. Without intending to, I found myself engaged in a defense of the value of Platforms, and the importance of providing one. The team liked it so much that we decided to reprint it, in edited form, here—I hope you will find it as interesting as they did.
My colleague Orran Krieger (https:// compute platform with its System 360 www.bu.edu/eng/profile/orran-krieger/) series (https://en.wikipedia.org/wiki/ of Boston University pointed out to me IBM_System/360). a couple weeks ago that the people IBM, in its heyday, was who really made money on the original the perfect platform Internet boom in the ’90s weren’t the company—their business software companies who built the was enabling businesses applications that people actually used. to compete, and they The real winners of the Internet gold built platforms that would rush were the networking companies, support any workload for whose work was defined by the success any customer. That there (or failure) of their implementation was value in this for IBM of the Internet Protocol. These was undeniable, but the real companies—first Honeywell, then more value was for the customers spectacularly HP and Cisco— provided they served, who could use the platform that everyone had to IBM’s platform to do until- have to be online. They could partner then-unimaginable things for with anyone to sell their gear and they their own customers and to could support any workload and any make their businesses more application. The enormous campus that effective and more efficient. Cisco built here in San Jose with all the I have some personal money they made providing the platform knowledge of this: My father- for the Internet, is evidence of the value in-law Ron Rowe built a a good platform provides. company called J.W. Pepper Extra points if your thoughts when (https://www.jwpepper.com) that reading the above turned to IBM, who relied on IBM minicomputers from the provided the ultimate (at the time) System 3 on to the AS400. In those
23 RESEARCH QUARTERLY
VOLUME 1:2
days, the IBM platform extended to A platform based entirely on open their sales and services organizations— source, where customers and partners my father-in-law’s IBM sales rep had could not only see how the system a desk in their office, and he spent his was working—they could contribute to time understanding their business and making it better. The success of this working out new ways IBM could help model, and of the broader open source them succeed. Of course, this made movement, has benefited not only end IBM a lot of money, but it was also an users but also vendors who provide the essential ingredient in the success of hardware our platform runs on and the J.W. Pepper. software that runs on top of it. Only the open source development model can It is worth noting that IBM’s platform Red Hat [succeeded] keep up with the rapid advances we’ve was just that—a thing that you built seen in the almost 20 years since Red by providing something something on top of. Prior IT vendors Hat Enterprise Linux first shipped. with a key difference... A who wanted to work with J.W. Pepper platform based entirely came in intending to build a monolithic, So, where does this leave Red Hat bespoke solution for them, and utterly Research? We aren’t engaged in on open source, where failed. IBM’s genius was in refusing to do computer vision, self-driving cars, customers and partners that. The way Ron tells it, when he asked or protein modeling. Instead, we are could not only see them to do it, they turned him down flat. focused on areas we think will make our Instead, they provided the platform and platform—not just RHEL but also the how the system was the training for J.W. Pepper to build its OpenShift container platform and the working—they could own IT competency to create custom services supporting it—a better, faster, contribute to making it software that was right for its own more capable platform for those who business. IBM maintained the platform are doing that work. Our mission is to do better. and Pepper maintained the business- the research that enables your mission, specific software that ran on it. With a whatever that is. programming competency far ahead of AUTHOR its competitors, J.W. Pepper went on to dominate the market for choral and band — Hugh Brock, Research Director sheet music for the next twenty years. for Red Hat
Today, Red Hat is a successful software platform company, having outlasted a host of other platform suppliers in the enterprise OS market. We arrived at this point by providing something with a key difference from all that had come before:
24 RESEARCH QUARTERLY
VOLUME 1:2
HOW A RIG IS BORN - BUILDING A RESEARCH INTEREST GROUP IN TEL AVIV A Red Hat Research Interest Group (RIG) is a self-organizing team that drives support for and participation in local technical research projects at academic and government agencies or industrial groups. How is a song born? Every kid in Israel is raised hearing a very driven. The different perspectives — famous Israeli children’s song written by create a nice blend of different ideas How is a song born? Yehonathan Gefen named “How A Song and perspectives. Most importantly, Is Born?” that explains to children how a the champions are not always the most Like laughter song is being created. professionally advanced or technical It starts from within people. Anyone that is keen to get things And then rolls out Following the music, I would like to done can “push” the initiative forward. How is a song born share some key points along with some examples that will give you a glimpse IDENTIFY THE WHY Like a baby In the beginning it hurts into the journey we have just started in Hopefully, the initiator of the first steps the Ra’anana Red Hat office forming knows why they are doing it. However, But then it comes out the local Red Hat Research Interest in order to recruit your champions And everyone is happy Group (RIG). You’ll see how the open and the rest of the group, the “why” Suddenly it walks on its own culture approach to starting grassroots has to be crystal clear. The key point How is a song born? initiatives works to hone the interests of is that defining the group’s purpose Like a baby a diverse group. and motivation, both as a group and as You put three words together FIND YOUR CHAMPIONS individuals, will help the group perform better in the long run. And warm is on a low heat Igniting an initiative is never easy, and You run quickly to bring even in Red Hat where the open culture For example Red Hat “as a company” is receptive to new initiatives, in order has an interest in creating research Some onion from the neighbor to get the momentum rolling you need groups in different locations in order You add two rhymes to form a coalition. So start small, talk to get exposure to local innovators A bit of pepper and salt to some people, sell your idea, listen and prove its interest in fostering new You mix in three sheep to them, then get their feedback, ideas by supporting research projects And throw in a cube of ice brainstorm with them, and together at all levels. The local Red Hat office, form a group that can lead the way. In on the other hand, might have a more Ra’anana, the champions of the RIG concrete goal of socializing with the are a combination of people that really “right” academic schools in order to care about patents and innovation create a stronger recruiting pipeline in and others that are more academically the future, or of developing a curriculum
25 RESEARCH QUARTERLY
VOLUME 1:2
that will enhance the technical skills resources and time. Start small, identify STEER THE GROUP of the candidates, or of stepping into a few partners, and invest in creating Engineers like to innovate, and they like the academic world from a business successful partnerships rather than to work on cool new things. New ideas perspective, and so on. The people spreading your resources across many will percolate as soon as you start rolling. forming the group may have totally partners. You just have to steer the group, work different motivations; self-focused JUMP INTO THE POOL in small steps, make sure the group is or altruistic. They might be using this working on a regular cadence, don’t over initiative as a ladder for developing their Once you have achieved the first three commit… and don’t stand in the way. own career. They might have a special steps, you have to start the ball rolling. interest in a specific school or project, Get the champions, have a few basic So, I can report to you that the Israel or maybe they just want to work on a first steps identified, create some buzz, RIG is alive and kicking. It is still in its research project. All these motivations and initiate the group. Once you have infancy, but we are already proud. As any are valid, as long as we are aware of jumped into the pool, and created the parent would teach their child on how to them and know how to unite them to group, it will have its own life, it will take small steps to create a song, we are drive the group forward. generate new ideas for the group to confident that the small steps we have evaluate and decide which ones to taken will produce satisfying results. IDENTIFY THE KEY PLAYERS pursue. Our professional world is changing AUTHOR From our experience in Ra’anana, the fast, and the academic world is as well. — Miki Kenneth, Research Program group came up with several ideas, Schools around the world are reaching Director, Office of the CTO, Red Hat discussed each one and decided to out and seeking industrial partners. pursue two different goals. One was to The trick is to identify key players try to attract more students to DevOps who are willing to partner and try and track—so one team is working both on form the right partnership. The right the marketing side of the problem as well partnership consists of good will, mutual as on the technical side of preparing the agreement on directions and interests, materials for a course for next semester. and “of course” technical depth and The other goal was to define a platform know-how from both partners. From (and a set of processes) that will create our local experience, we found that it an easy interaction between project is sometimes easier to partner with the ideas, academic partners, and technical Article links: “second tier” universities (not the three Yehonathan Gefen - https://en.wikipedia.org/wiki/ Red Hat mentors. People teamed up on best ranked ones), as these are more Yehonatan_Geffen these projects, according to their own open and receptive to new ideas. On How A Song Is Born? - https://lyricstranslate. special interests, and now there are two the other hand, do not commit to too com/en/%D7%90%D7%99%D7%9A- separate teams working in parallel. %D7%A9%D7%99%D7%A8- much. Partnership, is something that %D7%A0%D7%95%D7%9C%D7%93-eich-shir- you need to invest in, it takes energy, nolad-how-song-born.html
26 RESEARCH QUARTERLY
VOLUME 1:2
Red Hat Research Quarterly delivered to your digital
or physical mailbox? ABOUT RED HAT
Yes! Subscribe at research.redhat.com/quarterly. Red Hat is the world’s leading provider of open source software solutions, using a community- powered approach to provide reliable and high- performing cloud, Linux, middleware, storage, and virtualization technologies. Red Hat also offers award-winning support, training, and consulting services. As a connective hub in a global network of enterprises, partners, and open source communities, Red Hat helps create relevant, innovative technologies that liberate resources for growth and prepare customers for the future of IT.
NORTH AMERICA 1 888 REDHAT1
EUROPE, MIDDLE EAST, AND AFRICA 00800 7334 2835 [email protected]
ASIA PACIFIC +65 6490 4200 [email protected] LATIN AMERICA +54 11 4329 7300 [email protected] Feedback, comments or ideas? Is there something you’d like to read about? facebook.com/redhatinc Drop us a line: [email protected] @redhatnews linkedin.com/company/red-hat
27 RESEARCH QUARTERLY
VOLUME 1:2
COMING IN NOVEMBER: • Research Day 2019, part 2: Gordon Haff looks at unikernels, FPGAs, self-driving device drivers, and much more • Roll Your Own Processor: Building an open source toolchain for FPGA code • DevConf US round-up, plus fun party photos!
Bringing great research ideas into open source communities
August 2019 – Volume 2: Issue 1