Working With Upstream: An Interview with Laszlo Peter
Laszlo (Laca) Peter is a release engineer at Sun Microsystems working on Sun’s Desktop team. Stormy Peters interviewed Laszlo to learn more about how companies working with desktop technologies can work effectively with the upstream GNOME teams.
Laszlo, many companies have trouble working upstream. I understand that Sun made a big push a few years ago to work more effectively upstream and it’s been very successful. Can you describe the problem that Sun was having before you started your upstream push?
When we started to work on porting GNOME to Solaris, it always took us a good while to get to the point where we could build it and use it.
We needed lots of code changes. By the time we finished, a newer version of GNOME was out. The new version would be equally broken on Solaris and most of our code changes didn’t apply to the new version, so the whole process started again and we never seemed to catch up.
We realized that we needed to work with the GNOME developers to integrate our patches in the upstream code so we wouldn’t need to do this for each and every release.
Working upstream was good in theory, but in practice, our patches were all 6 months behind. Sending patches that were against 6-month-old code didn’t help much because the code had moved on and the maintainers were not interested in forward-porting our patches. So we realised we have to work with the latest and greatest community code and send patches as soon as we notice that the code breaks on Solaris.
What does it mean to push something upstream?
First of all, [it means] writing patches that are acceptable in the community code base (i.e. a fix for Solaris doesn’t break the code on other OSs) and then sending it to the maintainer/list/[bug tracker] as soon as possible, so they can easily incorporate it in the community code base.
What did you have to change internally? What did you have to do differently in order to work upstream?
We had to adopt our internal processes to make sure we didn’t start lagging behind the community code base again. We need to keep working with the latest code, reporting issues upstream as soon as we find them.
Catching up is much harder than keeping the pace. Just like upstream, we have two active branches of our desktop: the latest stable version goes into Solaris and goes to our customers, and we only fix the really nasty bugs in this branch. The development branch is what we concentrate our efforts on. We keep updating the development branch with the latest GNOME development tarballs as soon as they are released and each tarball has an internal owner who is responsible for updating internal patches when they break. If they do their jobs well and get their patches upstream, they have less work to do. That’s quite an incentive. 🙂
Another incentive for Sun developers to write good patches is that we require internal review of all patches that are not pushed upstream… if a patch is pushed upstream, we trust the maintainer’s opinion so it need not be reviewed internally.
Does that mean you have more people working on GNOME now? Do you have to have an internal developer for every external GNOME maintainer?
No. Like in the community, one person can maintain more than one package. Since our internal maintainers are only responsible for maintaining (or rather: reducing) the difference between the upstream code and Sun’s code, they are able to maintain many packages.
Was it a hard sell internally?
Not really. It makes a lot of sense to work this way.
Did pushing things upstream slow you down?
On the contrary. It made us more efficient than ever before. We manage more packages, keep them better updated and they work much better on our platform, out of the box.
Do all of your changes make it upstream? What happens when they don’t?
No, not all changes are accepted. There are different situations. The worst one is if a patch we submit is ignored. In that case, we don’t know if it’s acceptable or needs work. In some cases, the upstream maintainer has different ideas about a fix. We want our engineers to negotiate and rewrite their patches if needed, until they are accepted. Finally, in some cases the maintainer disagrees with the bug report itself and is not willing to accept the changes in any form. In this case, we need to re-evaluate the bug internally.
If we decide that we still want it, because our customers want it, we will keep the patch. These are patches that we really hate because they need to be merged with the upstream code changes all the time.
What types of things do you push upstream? Are these Sun specific features? Things just your customers have run into? Things you think would be cool?
Preferably everything. In the case of features, it’s a good idea to work with the upstream developers from day one and maybe develop in a branch of the upstream SCM [(source code management)].
Are you still able to get customers the products they need, when they need them and still work effectively upstream?
Yes. We can make the changes that the customers require in the stable desktop and not necessarily push that specific change upstream. At the same time, we come up with a long term solution for the problem and implement it in the latest development code.
Do you push out changes to customers and then push them upstream? Or do you push them upstream and make sure they are accepted before you use them in customer products?
We concentrate on the development version of the desktop and push most fixes upstream before the stable release. Patches that do not get accepted before the stable release will make it in the customer product first.
When someone in the company is new to open source, how do you educate them on how best to work with the upstream community?
Working with the community is part of the process, if you like, part of our job description. In my experience, most people get it quickly. Others may need a little push.
What advice would you give a company trying to work upstream?
Encourage your developers to communicate and work closely with the community. Send them to developer conferences and summits to meet them face to face. It’s much easier to work with someone when you can put a face to a name, and code changes are more easily accepted after a beer or three. 🙂
What could the community do better to help companies like Sun work effectively with upstream?
Review our patches in a timely manner. The worst feedback is no feedback.
For companies looking to work more closely upstream, can you give them an idea of how hard this was to implement? Did it take a lot of convincing? A lot of time up front? Are you saving a lot of time now?
When a company decides to use code from an open source project, they give up control over the changes and directions of the code base. They become part of the community and can influence decisions just like any other community member, but ultimately they lose control. Their best shot is to participate in the community and understand, discuss, influence design decisions.
However, the company still has full control of their internal processes. They can and have to define them such that they can provide what their customers want while they still go with the community and not against it. The initial implementation is likely to take significant effort, but they will save that later.
About the Author
Stormy Peters is the Executive Director of the GNOME Foundation. This is her first article for GNOME Journal.