Today it won’t be so much about me and the project stuff, but it will really be more about the technologies as we promised. Last time I expressed enough, I had enough space, so today I’ll leave it to my colleague.

From the project part, it is only necessary to say that the things we will talk about were partially based on the original concept and the technology stack, as we planned the project. With some historical knowledge and experience, we planned a solution that was built on one technology and then, actually, upuon the advice of the manufacturer, we chose a newer and more modern technology.

I don’t say the abbreviations on purpose, because I’ll admit that I’m not completely sure about them, so I’d rather leave it to my colleague so I don’t say something wrong here and I’d rather give him the word right away, and if necessary, I’ll step in somewhere and add things with regard to the impact on the project, when it had an effect on numbers, deadlines, etc.

For today we will have more in-depth topics about the chassis and networks. The first topic will be SAN (Storage Area Network) and we will deal with how we completely wired it, how we use it, what elements and ports we have and at what speeds.

As such, our storage has two meshes and each has two 100 Gbps ports. Everything is configured so that it is in high availability, that is, in the event that a port fails, the module is defective or something happens to the optics or there is a problem with the entire switch or network card, we have a second network, port or a switch that is capable of taking over the full load of the entire server 24/7 and the customer has no idea that we, the operator, have any problem with the network or any part of our infrastructure.

For this, we chose 100 Gbps lines, given that we have them for storage as such, and our computing servers are connected by 4×25 Gbps lines, which is sufficient for us at the moment. We have completely separated the data part from the storage part, so in the event that a server needs to access a drive a lot, it will not affect the throughput to the Internet in any way, even within individual virtual servers that may be located on other physical machines.

Why didn’t we choose EFC and go this way?

It’s one technology, so we don’t need to have other modules, meshes, we only have two switches and we have one complete solution, I don’t want to say from one manufacturer, but as far as networking is concerned, we have one vendor that we chose and thanks to that, in case we have a problem on the network, on the module, on the optics, on whatever, so I can take any other optical module that we have and just go click-click and we’re on our way. They don’t have to deal with what type of module we have, what kind of cable we have, what is actually there and from where to where what is connected. That’s why we chose this solution.

That is the storage part and the network as such. Thanks to the fact that we have 100 Gbps lines there, we decided to have the highest possible throughput and to load the CPU and RAM as little as possible, because we know that the more data throughput we have through the networks, then some means must serve them, which is processor. The more packets it handles and the greater the data throughput through individual ports, the higher the load. Based on this information we started to increase the MTU.

We increased them from the basic 1500 B to 9100 B and some change. This is the highest possible value of the switches we have used. That’s where our hell started, because it’s called what went wrong, so of course we immediately started to find out that not everything that should work beautifully does.

The servers worked beautifully on 9000 B, but only within the switch. So everything was fine within the switch, but in the case when we did burst tests and we disconnected half of the network and one part of the servers ran through one switch and one part through the other switch through the technology that I will mention later, we were not able to squeeze out MTU higher than 8 000 and we were still solving what and why, until we solved it with the manufacturer and variously in the knowledge base, what and how we forgot to configure where, since the basic solution was that we will drive everything 1500 B and not solve it. But in order to reduce the load on the servers as much as possible, we started increasing and slowly but surely we started to configure the switches. And where we thought that all the individual parts were needed, until we spent about two days, when we searched from morning to evening, where we lose the MTU and where it is missing, and it happens that the 9000 B it does not flow between switches, but only within one switch.

We found that out thanks to VXLAN, which brings me to the next technology. The first technology, as such, since we are used to it, we thought that we would build these two switches as a Virtual Chassis Fabric, that is, we would connect these two physical switches into a virtual one and configure it as we are used to. We have two physical boxes and I will connect to some management via one IP address and both boxes will make one virtual for me and I will have everything from the comfort of one terminal, I will configure everything from one line and I will not have to jump from left to right.

That did not happen. Thanks to the manufacturer’s recommendation, since we want to have the highest availability possible, preferably 100% uptime, he recommended EVPN technology to us to actually build an EVPN VXLAN Fabric from these boxes, so that in the future we can scale and not be limited by vendor lock-in or model lock-in because in the case of Virtual Chassis Fabric – it will certainly be the case with other manufacturers – you can only connect either the same models that support this technology, and not all models support it, or very similar model series. For example, if I have a 3300 model series, I can’t connect the 4400 series to the VCF because it simply doesn’t support it. It might be possible to connect it, but it would be very challenging for the manufacturer in terms of technology. Due to this, EVPN started to be implemented, saying that in this case all these problems will be removed, so now that we have two switches, we can buy any other switch that supports this technology and we can scale further. We are not limited by a specific model type, but we can buy any model type and actually from any other vendor and we can go further and we are not limited.

We try to keep all the technology in this way so that we are as little limited as possible.

As for the virtual chassis, EVPN and the MTU, since it is an EVPN VXLAN Fabric, VXLAN as such takes 50 B for its own overhead, which we did not know at the beginning, because it is a new technology for us. We set MTU 9016 everywhere at the beginning. Since we had one server on side A and one on side B, it should have passed through the VXLAN. Because it was there, we were missing 50 B in the data frame, which we found out later in the documentation, when we searched in the knowledge base and with help from the manufacturer, and we increased the MTU on the switches to the maximum possible, which for them is about 9206 B. With this the problem disappeared, so we can have 50 B for overhead VXLAN and subsequently also for VLAN and we still have the possibility to use 9000 B purely for data transfer between switches and from one server to another switch.

I would still go back to the endings from the past. Those of you who were here last time, we were dealing with unterminated cables and rotated network cards on Intel computes, and since we have EVPN Fabric, we no longer have Virtual Chassis Fabric, so in the end AE would behave the same with LACPI, only we would probably figure it out a little earlier.

In EVPN Fabric, we don’t have one virtual switch, but we have two physical ones, so when I need to configure something, I have to go to one switch, then to the other switch. On the one hand, it’s great, on the other, whatever I do on one, I have to go to the other and do a 1:1 configuration.

On the one hand it’s good because it’s separate, on the configuration part it’s more tedious because I have to do things twice. But it has far more advantages in terms of security and operability.

What kind?

In the event that I have a Virtual Chassis Fabric and EVPN, so if I take it to our Virtual Chassis Fabric where we have a switch, then when we were doing burst tests, since we had the switch in a weird state, I started to reboot all the switches between each other so that to the failure of other switches. Everything was supposed to look beautiful in such a way that I come to VCF, reboot one member and nothing happens. Only people, offices or  computers that are on that particular switch that reboots know there is an outage.

I did it 6 times and this chassis fell apart each time in such a way and to such a state that the whole thing had to be restarted anyway, because not all the ports were connected correctly, which was only discovered afterwards. It happened that we have 6 switches in this chassis – one has the permanent role of a master, the other is always in the role of a back-up. If anything happens to the master – for example, I manually reboot it or it loses power or anything else happens at the system or hardware level – and it does the virtual switch, the backup part takes over all its operational capability.

Unfortunately, it’s not immediate and it’s usually 1 minute. We have tested it to be 20 – 40 seconds. Such an outage under the cloud part is critical for us, because you cannot see what is happening and you have to deal with it immediately. We don’t want customers to suffer a 20 second outage because something happened somewhere.

EVPN solves all this, because each switch is stand-alone, and if something happens to one switch, EVPN on the other part is non-stop in an active role. Whenever anything comes to her, she processes it. It is not that she transfers the traffic to the other part, but the traffic that goes to A or B switch is processed by that specific switch and is either sent back to a server or sent to the Internet. So that’s all about virtual chassis and EVPN. This is why we chose EVPN VXLAN Fabric and not Virtual Chassis Fabric.

So there isn’t a delay before it decides that the master is switching to the slave?

Exactly. And before it completely takes over all activities, before it pours all the configuration that was on the master into those line card switches. The switch must learn all the MAC addresses at that point and start working.

Oh, I’m aiming for the next upgrade.

We have to deal with security patches and new features for the future. (It would probably be good). As for upgrades, within the Virtual Chassis specific versions support the so-called non-stop upgrade, with the fact that this chassis should be able to be divided into two halves. One is being upgraded, the other works normally. Then the original part switches the communication to the newly upgraded part and upgrades itself to the current version. The entire virtual chassis is then folded into one.

Unfortunately, not all versions support this feature. It’s a feature that some manufacturers have been pushing for a few years now. This is not a threat in the case of EVPN, because now when I decide to upgrade one switch, I just turn off the ports that I don’t want to be affected. When I’m going to upgrade, I don’t want the switch to take over full functionality when I turn it on. There could be a bug in the configuration, for example between major versions, there could be some change, we’ve been through this in the past when we’ve had port names change, that can happen. If someone doesn’t read the changelog – it’s good to read them – if they don’t think about this, anything can happen.

So it is important to check it. Of course – before I send the switch back to production again, it is necessary to check everything, and when I consider it appropriate that it is in order and capable of operation, I will connect it back to EVPN and I will again have two switches in active management and we will have load balancing.

That’s all for technologies as such.

Does anyone read the change logs?

Now it can be seen that this is necessary. And I have to say that our technicians have even started reading the manuals sometimes. We’ve sunk so low and come so far. It just becomes necessary. Those changes are fundamental, and what Vasek mentioned was quite a big point in technology, because when the names of the ports we use change between individual versions of the operating system, if you don’t know it in advance, you’ll go crazy.

All of a sudden, things that you don’t expect start popping up, and moreover, if you do it at the level of high technology availability, it will only change in half and not in the other half. So the N+ things  suddenly aren’t there, it’s called something else, so it makes him crazy.

This is exactly the reason to read the change logs and to see what changes are there. Despite the fact that those change logs also tell you if some of the technologies you use are affected by some potential bug. Those systemms should not be deployed the moment they come out, that’s just suicidal.

It is deployed after some time period and with some knowledge of what works and doesn’t work in that version and what technologies are affected. So it’s just not possible to shoot like that, a new version was released yesterday, so we’re going to pour it in there.

Do this only in a lab environment where you can afford to crash and test these individual new releases, because if this happens to you in production, if there’s a bug that could easily cause a kernel panic on your switch – what you’ll do remotely if you can’t do anything with the switch at that moment. You should really read the change logs.

The first thing is to read the change log, the second thing is to test it in the lab, to have the means, not to be a punk and not shoot it straight into production, which often happens, so of course it has some demands and you have to have the equipment available . Have a lab where you can try it out. It’s not cheap, it costs money, but it’s necessary, because if you then have to ensure operation with some adequate quality, then it simply has to be done.

And even if you go through it, test it and try it in the lab, you still don’t have a guarantee that there won’t be a problem in the production. But you will do your best to avoid it.

Last time we promised one more topic, and that is migration. We are talking here about the fact that we were building a new cloud and this went wrong, so let’s also look at one thing that went right.

The migration didn’t fail, the migration succeeded. I already partially said it here the last time. Not that it was without problems, those problems were caused by external circumstances rather than technical things. As I mentioned last time, from the planned dates and times that we had for it, for various external reasons, the time was reduced a lot and we were pushed to do it in an accelerated mode, which of course increased the pressure on the technicians with those marathons that I mentioned last time , that they didn’t greet me for a few days, but it was done, it was done, it was done. Partly because the techs were well prepared, partly because we have a good team and they’re willing to go above and beyond, so I always like to praise them and thank them for that.

And I won’t give them to anyone! They are mine! If by chance someone wanted to drag them away.

There was really no problem except for the time pressure that was there. What was planned, of course, was that there should have been no migration. As such, the project did not count with any migration at the beginning because, as we mentioned last time, we were building the whole system as an extension of the original capacities.

Over time, the set-up changed, the purpose changed, and in the end we ended up with a completely separate operating unit, so in the end it was a complete migration from one cloud/provider/data center/servers to a completely different one – technologically separated, configurationally independent.

The technologies that Vašek mentioned helped us in this, because we were able to build some kind of network connection there across the data centers and locations, because they were moving between two data centers between different cities. All of this helped us to be able to take customers from on-premises, where they were, and from the cloud, where they were, and move them to our cloud.

So, from their point of view, it was actually a shutdown of those individual servers only for the time of the actual transfer of data from one cloud environment to another. We were only limited by the network speed we were able to get between those data centers, but otherwise everything else remained functionally the same for them.

In essence, their environment changed just by the fact that they started using a new platform, a new technical solution from us, but operationally the system remained exactly the same, so unfortunately it was with a shutdown, but on the other hand, our customers supported us in this, so we prepared it in such a way that it limited their operation as little as possible and the migrations took place very quickly.

In the same way that we are able to quickly adapt customers and transfer them to us, in this way we are also able to shoot them out of us relatively easily. That’s part of it, because sometimes the migration is needed. The customer asks how he can get from us, how he can get the data from us once he joins us. Since we have a technological computing part separated from the data part, the configuration of the virtual machines is separate and does not affect the data that is stored in the storage. It just depends on how I want to carry them. They are basically images in QCOW2, and if he drags them over the network or transfers them to the storage where we pour it, or a VPN is connected and it is sent, we have no technical problem with the migration. Converting this format to another is not a problem. Just as we migrate customers from VMware or Hyper V, the disk format is simply converted, so the data can be converted back in the same way and it works anywhere else.

This was basically the only technical complication that we had to ensure the connection with the original provider in terms of networking. These new technologies gave us the tool to do that, they gave us the opportunity, so the migration was done in such a way that the data was transferred, so the downtime was for the time before the data overflowed from one environment to another. The customers in the finale accepted it and had no problem with it, it was planned to have minimal impact on them, so it mostly took place at night or on the weekend.

They just said, how can they have the given downtime, and we tried to adapt as much as possible.

Because of this, the guys didn’t get much sleep – as it was mostly at night and on weekends – but we haven’t had a single complaint from a customer that the migration would have been a problem.

So I don’t think it really belongs here.

But it’s good that we could end with it.

We certainly like to show off.

Tap yourself on the shoulder, you’re good at it.

Over time, yes. When it was over and it was fresh, the guys didn’t talk to me for a few days.

We didn’t want to talk to anyone at all because whoever came to us, it was a rush, a rush, of course it was late yesterday. But if there is such a bigger project, then it belongs to it, otherwise it wouldn’t be a project.

But they did well. And I said, it doesn’t belong here, because it didn’t go wrong, but went right.

Well, we won’t cut it out, since we’ve already had so much fun here, but after all these struggles, you have to tell yourself that the result is worth it.

We are happy with it. Technologically, we have moved on, we have improved, we have more control over it, it is our solution. We went from something that we had rented and hosted, we didn’t have full control over it. Now it’s our own solution, on our own iron, it opens up a lot of new possibilities in what we can do. It’s definitely a step we’ve been thinking about for a long time, we did it at a time that wasn’t favorable at all – we couldn’t have planned it. But we did it and today we are happy about it and we think that with hindsight it was the right step and that it is moving us forward. And we certainly do not evaluate it negatively, but we think that it opens the way for us towards further development and possibilities of what to actually do in the cloud, to offer to customers. It gives the boys new toys and new things that they can try in a different way. Also see, for example, upgrades and device management.

What we were actually dependent on someone else for, today we have full control. We can plan it, the guys can test it, they have the laboratory so that we can do everything professionally and not go into it head-on, but to do our best to ensure that during these changes and activities that have to be done so that the impact on customers is as small as possible, optimally so that they don’t know about it at all – in the laboratory they have all the equipment that we actually operate.

Which it almost did. Some downtime always has to be done.

Those upgrades have to be planned, we inform customers about it, but now, as the system is built, they should actually only know about it from those emails.

And only you should know about it.

Exactly, we who will be sitting there when something happens and what we will do about it.

And this should not happen thanks to the laboratory, of course. But we know how it is in the real world.

See also the question with change logos.

Exactly. A few days in advance to download change logs, to upgrade, to test configurations and then to production.

I’m thinking how to sum it up for the project part. As I said at the beginning, we didn’t fit in the time, we basically fit in the budget, even with the terrible changes and problems that were around it. This is actually positive. The fact that we didn’t fit in the time didn’t really matter so much in the final from an operational point of view. Of course, the pressure was greater on the boys, because some things that they had planned for a more reasonable time and dates, so that they had the opportunity to relax a little and not be at work almost non-stop and always awake, didn’t quite work out.

However, projects are always like that. There is no project where everything will go well, you have to expect that something will not go well and then it only depends on how you deal with it.

I think that all the things that happened to us led rather to the fact that the project has improved, that it is a better technical solution, the operation is more reasonable for us and more friendly for us, better, we have more control over it. We essentially have our own solution, so from this point of view, if we ignore the fact that the project was longer than planned, it actually only had positives for us. Although sometimes they were horror works.