Running on OpenBSD - The backend (Part 1)
In a world that three operating systems dominate (Windows, Linux, MacOS) and alternative sounds weird we gave a run at OpenBSD as our operating system from end-to-end. The following document is the first part of a four part paper that describes how we managed to setup our entire company using only OpenBSD and its provided ports and tools.
We hope you enjoy the reading.
When we tell our customers that we don't run any Windows on our company they get surprised. Its strange for them to even think about a world without Windows. This surprise almost always is followed by the questions "What do you use?" and "How do you do it...?". Echothrust Solutions runs and operated entirely on OpenBSD, the version follows current releases closely.
In the following paper we will outline some of the difficulties and some of the benefits we gain as a start-up business by using OpenBSD for all our business operations and how does that saved us thousands of pounds and our peace of mind.
Even though, many users will argue that OpenBSD lacks this and that, in our daily operations we haven’t encountered a problem or limitation that the developers and port maintainers haven’t provided an alternative. Sure, we might not used all those fancy interfaces but this did not stopped us from producing the desired results and offering automation in our methods that outperforms many others.
For Intranet, development and other servers where HTTP service was required we used the default apache shipped by each OpenBSD release. Apache was loaded with mod_gzip and PHP in order to run the applications used.
For our PHP we loaded numerous extensions all of which were provided by the OpenBSD package collection.
- Multi-Byte String support
Depending on the server some extensions where not loaded since there was no need for them. For instance only the development server used the mhash extension for some applications that were under development which utilised this feature.
Our mail server was one of the most complex services, but the process of setting it up was quite easy and smooth with the great work OpenBSD did on the provided system utilities and packages.
Our mail server of choice was Postfix, it is one of the very few occasions where we go out of our way and not use a default OpenBSD service. However years of using Postfix provided us with a very good knowledge on configuring the service and we were not willing to trade. Mostly personal preference than functional you may say.
The mail server had installed Amavis, Clam Anti-virus and postgray, which in combination of spamd provided by OpenBSD, allowed us to have a spam free mailbox. Although the delay caused by spamd was a bit tiresome at the beginning, after adding a list of known mail servers we pretty much were fine. We also added outgoing mail servers that we contacted (by sending a mail lets say) to our allowed list and this allowed for smoother reply. However in cases where the receiving mail server was not the same as the sending, we had a couple of phone calls about it, but once we explained to our customers that everything was working as it was supposed to, they were pretty happy with the initial wait, specially when we told them that it was one time in every 3-4 months or so.
Our mail servers also used encryption, with the help of our local Certification Authority, we were able to provide encryption all the way. Furthermore, we configured Postifix in such a way that it was always trying to negotiate TLS with outgoing servers (quite surprisingly many supported it) and thus provide further protection. Overall it was pretty nice but like we said very complex.
Since we had no Windows systems our file server was mainly supporting NFS. However since we could not exclude the case of a visitor system (sales laptops let say) we used samba as well for our MacOS and Windows visitors.
The systems that needed to access the file server through NFS, things were a bit more strict since all the NFS traffic had to go through a encrypted tunnel thanks to the magic of ipsecctl.
No need to mention how we did our firewalling I suppose, but here it is. We used PF and utilised most of its features in different occasions. Anchors, tables, concurrency limits, depending on the use and policy for the particular firewall. Some policies were more loose than others. Different rules for the development network, different for the gateway and such.
It is worth mentioning that tables and more recently anchors allowed us to maintain and manage our firewalls with such ease . Scripts and tools fitted hand and glove with pfctl.
Furthermore, we run PF on all our servers and most of our in-house workstations for maximum security, the lightweight and fast implementation of PF make it transparent to the system's overall performance.
For remote offices and roaming access to the data, we used isakmpd. Although I would lie if I said this was an easy task for the novice, however with a bit of good reading of the manual pages and the many documents available on the net, it can be done painlessly. We used x.508 certificates for our authentication between VPN nodes and Roaming users and it was obvious we needed a lot of certificates and lot of time.
For our backup server we created a couple of shell scripts using the Korn Shell provided by OpenBSD. The backups were incremental since size was of the essence.
After all backups were finished for the day we scanned the changes for unauthorised ones or changes in danger zone directories such as executables. Since the backup was incremental this operation did not take much space. The backup scripts were run from the server using crontab entries and accessed each server through customised OpenSSH subsystems.
However, taking a backup is only half solution, you have to make sure you can also restore these backups. Luckily for us everything worked perfectly and we were able to restore systems to any level we wanted. We could back-roll backups from previous dates. Everything worked so smoothly we almost forgot that the service existed.
Since our network was quite complex and with the additions of networks which were introduced through our VPN, it became quite tiresome to do it manually. However this did not last long as soon we used the ospfd shipped with every OpenBSD and all our troubles were solved with 4 lines of configuration file, it was simply brilliant.
We also considered the BGP alternative, also provided with every OpenBSD release, but our network topology did not require it. Still the option is there when need arises.
For our project management system we used the excellent DotProject which was one of the few applications we had to download manually.
However, we did encounter some problems with this scenario but we managed to overcome most of the problems with a couple of hours of PHP hacking. DotProject had some difficulties with PHP5 and MySQL 5.0 back in the days so certain modifications had to be done in order to make it behave.
The only regret we have from this adventure is that our hacks were so quick and dirty that we never managed to provide then to the developers. Soon the development of DotProject moved on, we did not follow the normal update schedule, we were quite happy with that version.
Our main word processor is a vi and we write all our documents in LaTeX, this made it difficult to share since there were occasions where more than one user wanted to work on a single document and through a web interface it was the only way of access. For this reason we used the DokuWiki. Although multiple wiki applications exist, some of them offer LaTeX exports and other similar usefull features, we found them so complex to work with that we simply could not stand to use them. However DokuWiki provided us with a nice twist, all the documents you work can be found on a very familiar structure on the filesystem as text files in raw Wiki syntax. So at the beginning we used to mainly work on DokuWiki and wrote a series of small shell scripts to transform the files into LaTeX for our PDF generation and any last minute modifications on the layout.
Later on, we created again a quick and dirty plug-in for DokuWiki which allowed us to export in LaTeX with the use of a button and with extra powers of cut&paste manage to do it much faster. Furthermore, DokuWiki offered us a great spell checker, even though we always double checked the documents with aspell from the ports.
Customer Relations Software
Our Customer Relations Management (CRM) software we used yet again one of the few downloaded manually applications, the phpBMS. This provided us with the ability to have an inventory for invoices, products, services, contracts, customer contacts all through a neat and tight interface.
For cases were limited access was provided we used the Squirrel Mail from the packages collection. The server used a certificate generated from our Certification Authority and thus all communications were encrypted end to end.
Later on this was replaced with a much simpler and much nicer (eye candy wise) package of RoundCube. This provided a much more familiar interface to non technical users.
As a side bonus from the need of the VPN concentrator, we had to research on the available (we knew some existed) Open Source projects. One project that stood out was OpenCA (https://www.openca.org/). We had to make a couple of small modifications and execute a couple of steps manually in order for the installation to complete. However since then OpenCA has released many versions and the instructions might not apply. As soon as we test an updated package we will post the updates or even create a port for OpenBSD.
End of Part 1
And that is our Part 1 of the four part series of Running on OpenBSD. Overall, OpenBSD as a server platform behaved impressively, considering the extra layers of security that are already implemented. Most of our core servers used software that was shipped by every OpenBSD release or packages.
However, for all packages that we believed were dangerous we always performed our internal audit which did prove beneficiary, we manage to plug a couple of holes and gain a better understanding of the used software. Packages like postfix did not pose thread, however PHP applications were particularly difficult to work on our restricted environments, which some times exposed vulnerabilities and bugs that had to be addressed. If for instance an PHP application was designed to work with register_globals=on unexpected things happened, this kind of staff.
One of the reasons we chose OpenBSD was security so it seemed as a waste if we installed applications that were untrusted, even though most of them were only available to company personnel only.
Only one thing is left to say Thank you OpenBSD team and all those package maintainers and testers out there for providing an exceptional system for servers.