Automatic system installation

Designed and implemented an Ansible-based framework that automated the full lifecycle of Debian systems across physical, virtual, and cloud environments, improving deployment consistency and reducing manual provisioning effort.

First, the framework allowed entire deployment of new Debian systems from scratch, hardware or virtual, without any user input. The whole system definition, as well as disks layouts were fully defined using simple YAML configuration.

A completely offline installation on physical systems was also automated. By using the servers management interface, to upload a custom Debian iso image and starts the system installation remotely. The whole process was non-interactive.

Systems provisioning

The framework organised a clear separation between the infrastructure definition and our ansible roles library. With the infrastructure defined in one place, for physical servers, virtual machines and cloud instances.

The ansible roles were defined in a different repository, totally decoupled from the infrastructure information, enforcing a genericity to any target type (hardware, virtual machine or cloud instance.).

From the command line, we were able to run one or multiple roles, on one or multiple systems. The framework allowed installation, uninstallation, upgrade as well as dry-run, testing and reporting.

Credentials encryption

All the credentials were stored in a separate repository, using GPG encryption amongst the team. This allowed smooth integration with other repositories, granular access control, and better credentials readability than Ansible Vault.

An abstraction layer was used, which allowed the use of different credentials storing system per environment, without affecting the roles logic. For instance, on-premises used locally GPG encrypted passwords (pass), and AWS environment could use SSM.

Custom reports

For auditing or reporting, custom PDF reports were generated and sent by email or stored on a Samba server. Reports included potential changes or current state, with various level of details and formatting.

Technologies

• Ansible, Ansible linting, Yaml linting and schema validation.
• Custom ansible dynamic inventory in Python.
• GPG with custom scripts for onboarding / expired / revoked keys.
• Credentials storage: pass password store.
• Shellchek and Python linting in Git hooks.
• Operating systems: Mostly Debian, and some CentOS nodes.
• Jenkins and Gitlab for continuous integration.

Bootstrapping

Using the framework above, we have deployed two datacentres, starting by the external and internal firewalls and the hypervisors, finishing with the virtual machines.

Security

For each datacentre, we used redundant external firewalls, connection tracking synchronisation, dynamic routing protocol (BGP) and transparent proxies with traffic interception and whitelisted domains.

Sensitive traffic that could be used for information leaking, like DNS and NTP, were intercepted and redirected to the internal servers.

Finally, Suricata IDS was used, with evebox for visualisation and custom Prometheus metrics.

Client whitelisting

As the legacy systems were using client IP addresses whitelisting, developed the entire system allowing the applications to update the firewalls whitelist in a secure manner. All the whitelists were synchronised and monitored using Prometheus metrics and OpsGenie alerts.

Authentication

For the servers access, users and groups authentication and authorization was linked to Office 365, using the LDAP protocol. The SSH keys were directly stored in the directory.

Monitoring

Built datacentre-wide monitoring and alerting with Prometheus and Grafana, increasing visibility across critical infrastructure and strengthening response to operational and security events.

Configured and tuned sensitive services, under AppArmor security profiles, and set-up OpsGenie alerts on security violations events.

Both internal and external firewalls successfully passed regular penetration tests and security audits. PDF monthly reports were generated with network rules and proxy accesses.

On the most sensitive systems, a small agent was installed to upload the logs onto AWS, using the S3 storage system.

Technologies

• Mostly Ansible, and custom modules or shell scripts.
• Operating systems: Debian.
• Debian preseed with simple YAML configuration, Jinja templating.
• Prometheus blackbox exporter with custom metrics written in Python or Shell.
• Authentication and authorisation using RedHat SSSD.
• Webs site fitlering with Squid whitelisting, using SNI only (no https inspection).
• Custom OpsGenie alerts using OpsGenie API
• Internal DNS servers with Unbound for caching, and PowerDNS with customized replication and SQLite storage backend.
• Client whitelisting with nftables, chroot’ed SFTP server, and inotify kernel daemon.
• Reporting: Pandoc and custom LaTeX code for PDF export, and Samba / postfix.

For the remote workers, deployed an integrated VPN solution, using both on-premise and cloud instances.

Four redundant access points, with low level packets authentication, and multiple factors authentication for clients.

Onboarding directly used the information from on premise Active Directory servers, to send the instructions and the second factor authentication by email.

Technologies

• Servers: Debian on Linode on op premise.
• Wireguard for site-to-site VPN between on-premise and cloud servers.
• Windows VPN Client: OpenVPN, with multi-factor authentication.
• Deployment: Ansible for both Linux and Windows systems, with WimRM authentication for the latter.

I used the same Ansible framework to deploy the platform on AWS, with little or no modifications. The platform was behind a load balancer and this time publicly accessible. Various strategies were employed to secure the platform, like AWS security groups and NACLs.

I also implemented a platform self decryption on boot, to protect both intellectual property and unauthorised access. The platform was protected by a randomly generated code generated offline, with a mandatory online validation method.

Bulbthings developed an asset management application, to handle their life cycle efficiently, and reduce the costs. Any type of asset was possible, like cars, printers, phones, etc.

The application targeted companies of any size, from small and medium-sized companies to corporations with international branches.

The initial application was a standard architecture, using a MySQL database, a PHP/Symfony backend with as a REST server, and an AngularJS/Metronic front end. The backend was complemented by other programs for HTML web sockets, in Go.

I took part in rewriting the whole application stack, with robust continuous integration techniques and bottom-up vertical development. Used the advanced PostgreSQL database features, like querying hierarchical data trees, dynamic schema updates, as well as geolocation and distance calculation.

For the back-end, we used server side JavaScript (NodeJS), with a REST API framework and web socket implementation out of the box.

The whole stack was thoroughly checked using a Continuous integration system, at many levels:

• For the backend: unit tests and integration tests with coverage analysis.
• Asynchronous testing of the Rest API on each commit.
• For the front-end: automated acceptance tests using two real web browsers and automatic screenshots.

The instances were automatically deployed by Jenkins on the GCE, using Ansible playbooks.

Each instance was automatically deployed on a subdomain and instantly accessible, without having to wait for DNS propagation.

We were using wildcard certificate to ensure simple HTTPS support, as well as http version 2 to reduce the load.

We also used four front-ends to distribute the load using DNS round-robin.

After working with a more and more Microsoft oriented environment, I have chosen this position as a “return to open-source fundamentals”. There was in this company a real desire to leave Microsoft technologies for less vendor lock-in solutions. My experience in both domains has been a decisive criterion for this role. I have managed to migrate all the services to Ubuntu and Debian based environments.

It was a challenging position, with a huge workload. The environment itself was built upon a ten years old structure, using mixed open source and proprietary technologies.

All these applications have been gradually replaced by modern LAMP stacks using agile methodologies, continuous integration and unit testing.

I have first installed a replicated and shared authentication system, so everyone had to remember only one password. I have then replaced AD by a Samba server, and activated roaming profiles.

The Exchange server has been replaced by three email servers hosted on two different internet connections. Emails were digitally signed and certified. A dual internal / external Jabber server was also included.

I have also deployed solutions allowing remote workers to transparently access all the services from home, like a VPN and a private cloud.

Finally, I have replaced in-house developed intranet with a secure and modern wiki.

The “Consumer Directory” is a web site containing a large base of consumers, who can register themselves. Once registered, they receive invitations to take part in surveys, after filling screeners.

I have used a Drupal as a content management framework to build a feature rich, modern version, and migrated transparently all the members. All their details have been validated and reformatted, and their initial password kept.

One of the prominent features was geolocation, allowing a simpler search than just by post codes.

The system has been customised to do mass messaging and automatic bounces management. All emails sent were digitally signed with DKIM and SPF, reducing false classification as SPAM.

Finally, all this stack was checked using continuous integration tests in a real web browser.

The company was initially using a Microsoft Access database to store their recruiters details, and was using paper mail to send invoices.

Using the same C.M.F., I have created an online system, imported the fieldworkers database, and added some useful features:

• Fieldworkers were able to update their details and upload their ID themselves.
• Emails alerts were automatically sent as soon as new jobs were posted.
• Project managers were able to search fieldworkers using various criteria, like domains of expertise, geolocation or personal data.
• Recruiters were able to submit their invoice directly to the system.
• The invoicing process was integrated into a workflow, involving project managers and finance department.
• Each action was logged and emails were sent on specific events.
• Project managers were able to rate recruiters or log misbehaviour. This was then accessible by the rest of the team.
• Statistics and Financial reports by month, recruiter, project, etc. with email alerts on budget overrun.

The authentication system was transparently integrated with our internal directory server, allowing the project managers to authenticate with their normal password.

To build the new web site, I have used the same framework, but this time as a Content Management System. With a customised responsive theme for mobile devices, the same information was presented differently on phones, tablets and screens.

Using a CMS allowed some departments to add content themselves, which was then validated and aggregated to create new pages. The web site was connected with an analytic system, with weekly and monthly reports sent by email to client services and directors.

I have also integrated a real time chat system, connected to the client services department.

Some custom modules have been developed, for instance to handle mailing lists unsubscriptions from clients.

Phone interviews from the call centre were manually archived and processed by an operator every day. I have replaced this by a set of Perl scripts thoroughly tested. Compared to manual processing, this was faster and less prone to errors.

The script was able to combine related records together and encode them in different formats, according to client's preferences.

The system was also able to upload records on client's web sites via FTP or to archive them on blu-ray discs at the end of each month.

Finally, an email was generated every night with a summary of events.

This web site has been developed to query national vehicle registration services in UK, to obtain all details from a plate number. The information returned was the make and model, the power, etc.