Thursday, November 7, 2013

Exchange 2013 CU2 redirection of OWA from HTTP to HTTPS

There's a lot of mis-information out there on the net right now about how to properly do redirection of the Outlook Web Access URL from an HTTP request to an HTTPS request. Most of this is due to how Exchange and OWA have changed over the past few versions... but some of this confusion comes from Microsoft itself, and their Technet articles! If you check out the Technet article "Simplify the Outlook Web App URL" at http://technet.microsoft.com/en-us/library/aa998359%28v=exchg.150%29.aspx, the article guides you through modifying the SSL settings, changing the HTTP redirect option, then going back through and cleaning up the mess you made in IIS due to properties inheritance on sub directories of the default website. What's worse, it doesn't even work!

 As it turns out, this process is actually just a copy/paste from the Exchange 2010 process: http://social.technet.microsoft.com/wiki/contents/articles/921.simplify-the-outlook-web-app-url-in-exchange-server-2010.aspx

So... what's the right way to do this? A custom error page! It's so much simpler than all of the mess above, and the changes that are made to IIS7 to make it happen are much less intrusive, requiring no cleanup or anything.

1) First of all, log into your Exchange 2013 CU2 CAS server and open IIS7.

2) In the IIS7 Management Console, expand sites and select the Default Web Site (not the Exchange Back End site!)

3) In the center area, double click on the Error Pages icon, then on the right hand side under Actions click on "Add".

4) In the Add Custom Error Page window, under status code, enter 403.4. For the response action, select the third option "Respond with a 302 redirect" and under the Absolute URL field enter your full url, such as https://email.domain.com/owa. Don't forget the /owa on the end! Click OK to close the window.

5) Open up a command prompt and type "iisreset" to apply the changes.

That's it! Now try browsing to http://email.domain.com and validate that you are indeed redirected to https://email.domain.com/owa. So much easier! The other real benefit is that you can leave the "require SSL" box checked on the default website and OWA subdirectory using this process, and you don't have to mess with the HTTP redirect option, which has the tendency to break all of your subdirectories by applying the redirection to them as well!

Saturday, April 20, 2013

Port Address Translation and NAT in Cisco ASA 9.1

I spent way too much time on this issue not to make an entry about it... hopefully this will save someone else the 12 hours of my life that Cisco snatched away from me with this little gem.

I recently had the (dis)pleasure of upgrading an older Cisco ASA 5510 from ASA 8.3 to the latest release... ASA 9.1. The system was stuck several revisions behind due to the memory limitations they imposed after 8.3, which required adding 1GB of memory to the system. The upgrade went well enough, no errors and internet access worked fine on first boot... however... it was quickly noted that outgoing email wasn't working. A bit of poking around revealed that the customer's spam filtering smart host, which was situated in the DMZ, was not sending email over the IP it was NAT'ed to... it was sending it out over the interface IP instead, causing SPF failures and mail rejection.

I spent hours building and rebuilding the NAT (which is completely different from the syntax in 8.3 mind you, so there was some learning curve here)... the lack of documentation online for our particular situation only exacerbated the problem. The key here was that the customer had a single external IP address that was previously responsible for both SMTP and HTTPS traffic... however... they had two internal hosts, a spam filter for SMTP in the DMZ, and an Exchange server for HTTPS in the LAN. In 8.3, it was a simple matter of doing a dynamic PAT for each host, assigning the IP that we wanted to NAT to and specifying the port it should use, with a matching ACL. Not so in 9.1!

Anyway, I'll spare you the endless futile attempts to figure out what eventually worked... and jump right into the code. First, I had to create TWO objects for each host that I wanted to have a PAT to (you'll see why in a minute):

object network Barracuda 
   host 192.168.12.3
object network Exchange 
   host 192.168.11.11 
object network Exchange-SPAT 
   host 192.168.11.11 
object network Barracuda-SPAT 
  host 192.168.12.3
 

Next, I set up my access control lists:

access-list outside_in extended permit tcp any4 object Exchange eq https
access-list outside_in extended permit tcp any4 object Barracuda eq smtp
access-list DMZ_access_in extended permit tcp object Barracuda object Exchange eq smtp

 Finally, I created my NATs:
!
object network Barracuda
nat (DMZ,OUTSIDE) dynamic ExchangeExtIP
object network Exchange
nat (INSIDE,OUTSIDE) dynamic ExchangeExtIP
object network Exchange-SPAT
nat (INSIDE,OUTSIDE) static ExchangeExtIP service tcp https https
object network Barracuda-SPAT
nat (DMZ,OUTSIDE) static ExchangeExtIP service tcp smtp smtp
nat (INSIDE,OUTSIDE) after-auto source dynamic any interface
nat (DMZ,OUTSIDE) after-auto source dynamic any interface


So... see what I did there? In the NAT statements, I ended up having to NAT the ip of each server TWICE... once as dynamic to the external IP address I wanted it to be NAT'ed to in the first two nat statements, and then again to the same ip as the first NAT, except this time specifying the incoming port for each server as static. Clear as mud? Yeah, same for me.

Apparently, this causes the incoming traffic to utilize the second two NAT statements (effectively the PAT statement from the 8.3), but the other two regular NAT statements are required to have the machines use the same IP for outgoing traffic.

The final two NAT statements above are just my regular interface dNAT entries, telling all other LAN and DMZ devices to utilize the interface IP for outgoing traffic.

Hopefully this helps someone else out there!

Friday, September 7, 2012

2008 R2 Domain Controllers and Scanning to a Folder

I am really surprised I haven't run into this before... but I guess there is a weird bug with many copiers out there that prevents scan to folder from working correctly under the following circumstances:

1) The source copier is using domain authentication
2) The source copier is trying to scan to a folder
3) The destination shared folder is on a Windows 2008 or 2008 R2 server
4) The destination server is also a domain controller
5) The destination shared folder is shared using 'authenticated users' as it's share permission, rather than specific users or the 'Everyone' group.

When these conditions are met, the scanner seems unable to scan to a folder. I found the two fixes so far:

1) Change the 'share' level permissions to 'Everyone' and 'full access'. This allows the copier to send files without authentication, thus bypassing the problem.

2) Change the user specified in the copier as the authentication account from a DOMAIN\Username or just 'Username' to their UPN format... Username@domain.suffix.

Either of those two changes seem to do the trick!

Friday, August 24, 2012

Installing and Configuring APC PowerChute Network Shutdown 3.0.1 for ESXi 5.0

I wrote this article original for ESXi 4.1 and PCNS 3.0.0, but enough has changed that I figured it warranted a new article... enjoy!

Shutting down windows servers with the APC network shutdown software is a no-brainer... but what about virtual machines? Sure, you could install the network shutdown software on every virtual machine, but that would be wasteful. Fortunately, there's a centralized way that you can shut down your virtual guests AND hosts... and the best part? It's (mostly) free! All you need is an APC battery backup unit, an APC network shutdown card, and a bunch of software. Here's what you need to do:

1) Download stuff. There's a bunch of it!
  1. Download PCNS 3.0.1 from the APC website. It's free now, and you do not even need to register for it (finally, somebody gets it!). You can find it here: http://www.apc.com/tools/download/software_comp.cfm?sw_sku=SFPCNS301&id=127&swfam=127
  2. While you are there, grab the latest firmware for the APC card you are using. Make sure you know the kind of network management card you have, as the older ones are not able to use the latest firmware.
  3. Download the latest VMware vMA from the VMware.com website under 'tools'. As of this writing, it's 5.0.0.2-724898.
  4. Grab the free version of Veeam fastscp... it makes loading the APC installer on the vMA a snap. Note that fastscp is now bundled into the full backup and recovery tool... so grab that.

2) Install the latest network management card firmware first. It may take several shots at it... for some reason it likes to fail, but doing it over and over usually gets it going. Don't ask why.

3) Create one vMA virtual machine by extracting that zip file you downloaded and then launching the vSphere client and going to File -> Deploy OVF Template. It will guide you through what's required, but you basically need to pick a datastore for its 5GB volume. I usually make the volume 10GB for snapshot space. There are options to give it an IP, but i found that it didn't end up mattering due to the following error when trying to power on the newly created vMA:

  • The only workaround I found for this error was to edit the newly created vMA, go to the Options tab, and under vApp Options select "Disabled". You'll get a nag screen indicating it is removing properties, but the only options that I could find set were items about DHCP, which we aren't using...

4) Open the console of the vMA, power it on, and follow the initial setup instructions. I recommend assigning a static IP as well as a real hostname to the vMA for use later. For the hostname, specify a full domain name such as VMA.domain.local. Once you are completed, you should be dropped to a blue and grey login screen.

5) Install Veeam FastSCP, then open it. Click "Add Server" and add the vMA server as a "Linux Server". Be sure to use the vi-admin username and password you specified while setting up the vMA. Uncheck the box to "Elevate account to root" as we do not have root access - it's been disabled.

6) In FastSCP, browse to the pcns300ESXi.tar.gz file you downloaded earler and right click / copy it. Note that you have to right click / copy in FastSCP, not in windows explorer... FastSCP does not have windows clipboard access built in. Once copied, expand your vMA in FastSCP, browse to the tmp folder, and paste it in the root of tmp. You can close FastSCP now.

7) Back in the vMA console, choose the option to "login", enter the vi-admin credentials, and do the following:
  1. Browse to the tmp directory (for linux noobs, type cd /tmp) and run "gunzip pcns301ESXi.tar.gz" to unzip the file.
  2. In the same directory, run "tar -xf pcns300ESXi.tar" to extract the file.
  3. Browse into the newly created "ESXi" folder and run "sudo ./install_en.sh" to start the installation of pcns 3.0.0.
  4. You will likely get prompted with a warning (read it, it's funny) and need to enter your password. After that, installation starts.
  5. Accept the license agreement, and accept mostly defaults. When it gets to the part about java, be sure to let the installer install it's own bundled version of java.  
  6. When you get to the part about entering an IP, just press "q" to skip it.
  7. Make sure you get the message about Installation has completed, and the note about how to access it - this means installation was successful.
8) Now, you need to add your ESXi servers to the 'fast pass' access via the following command: "sudo vifp addserver hostIPaddress". You will be asked to enter the password for your host, so vMA will have it on file. Do this for each host in the order you would like them shut down.
  • IMPORTANT: Make sure that the vMA is located on the last host you add to the fastpass list. This makes sure that your vMA is one of the last things shut down, and that it is able to give your final host the sutdown command, which will in turn shut down the vMA. You'll probably want to disable vMotion for the vMA too, or only map it's storage to the final host, so that it cannot be moved.
9) Now, we need to configure PCNS. Open a web browser and go to https://IPofVMA:6547 and follow the configuration wizard. The only step worth menitoning is that you should NOT check the box for "Turn off the UPS after shutdown finishes." If you check it, there's a good chance the UPS will turn off while your hosts are still shutting down. The caveat to leaving this unchecked is that your ESXi servers most likely will not turn back on if power is restored before the UPS battery loses all of it's charge... you'll have to use iLO or DRAC to get in and turn your hosts on instead.

10) Next, you will want to configure shutdown events. Once the PCNS wizard finishes it forwards you to the PCNS configuration page. Click on "Configure Events" and configure some of the important events like "UPS: On Battery" and "Input Power: Restored". You'll most likely want to notify users for most of the events. Don't forget to check the box for "Shut Down System" next to "UPS: On Battery" and set it to go off after a reasonable amount of time on battery (this depends on how much runtime your battery has, but I usually set mine for 5 minutes. If the power is off for 5 minutes, it's most likely going to be off an awful lot longer.)

11) Also, you'll want to check the "Connected Servers" tab on the left under "UPS information" to make sure your vMA IP address is listed. If not, you'll want to add it as a client on your UPS's network management card's web page.

12) If you are using VMware HA... you have a few more steps to follow in order to get your VMs to shut down cleanly. See the following article for more details and a script that you should add to your battery event: http://nam-en.apc.com/app/answers/detail/a_id/11622/related/1/session/L2F2LzEvdGltZS8xMzQ1ODI3Njg1L3NpZC9fZG9xUXY0bA%3D%3D

In short, it has you upload the following script to your vMA, then add it as a shutdown option:

shutdownvms.sh:
#!/bin/sh
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/vmware/vma/lib64:/opt/vmware/vma/lib
export LD_LIBRARY_PATH
export PERL_LWP_SSL_VERIFY_HOSTNAME=0
SAVEIFS=$IFS
IFS=$(echo -en "\n\b")
hosts=(10.216.252.167 10.216.252.168)
ups_vm="vSphere Management Assistant \(vMA\)"
for host in ${hosts[@]}; do
echo $host
source /opt/vmware/vma/bin/vifptarget -s $host
for i in `vmware-cmd -l --username xxxxx --password xxxxx`; do
if [ `vmware-cmd $i getstate | egrep -c "on"` -eq 1 ]; then
echo $i
if [ `echo $i | egrep -c $ups_vm` -eq 1 ]; then
echo "Skip shut down of VMA"
else
echo "Shutting down $i"
vmware-cmd "$i" stop soft
fi
fi
done
source /opt/vmware/vma/bin/vifptarget -c
done
 
I completed this by downloading the script file they had precreated, then used Veeam FastSCP to upload the file to the /tmp directory. Once there, I changed the permissions as indicated in the instructions (sudo chmod +x shutdownvms.sh), and followed that up by moving the file to the /opt/APC folder, just so I didn't forget what it was for. After that, I just went into the "Configure Shutdown" page on the PowerChute network Shutdown web interface and entered the script and it's path in the "Run this command" box. I gave my system 300 seconds to shut everything down, which seems to be sufficient.

13) If you are not using HA, in the vSphere client, make sure all of your ESXi hosts have their "Virtual Machine Startup / Shutdown" options configured and they are set to "Enabled", otherwise your guests are likely to just be turned off rather than shut down! (Note, this doesn't apply to HA configurations, as HA will keep disabling your shutdown options.)


That should get the system going! You can test it by, well, pulling the plug! If that's too scary for you, you can also configure the shutdown option for "PowerChute cannot communicate with the NMC", then pull the network cable from the network management card. If your hosts and guests shut down correctly, all is well!

Note: If your hosts still refuse to shut down, check out this article for a probable fix: http://nam-en.apc.com/app/answers/detail/a_id/11621/related/1. The long and short of the article is that you need to add the following line of code to the /opt/APC/PowerChute/group1/bin/shutdown file right after the line that says "export LD_LIBRARY_PATH":
export PERL_LWP_SSL_VERIFY_HOSTNAME=0
For linux noobs... you can edit the file by giving the command "sudo vi pathAndFileName", then pressing "i" to "insert" text... then edit the file like normal. Once you are done, press escape, then type ":" and "wq" then hit enter. Simple fix.

Sunday, August 19, 2012

Installing Dell Openmanage on an ESXi Host

Found this when paroozing the VMware communities sites for more info on Dell Openmanage for ESX.


Section 1: Install the vSphere Client and put the target host in maintenance mode

1. Connect to your Host Server via Internet address.
2. Download the vSphere Client appropriate for your version of ESXi
3. Install the vSphere Client
4. Open the vSphere Client and connect to your vCenter or directly to the host
5. Shut down all the guests on the target host, or migrate them to another host.
6. Right click on the target host in the left hand pane and click "Enter Maintenance Mode."

Section 2: Install Dell OpenManage

1. Download and install ‘vSphere CLI’ from VMware website (you will need to create an account in order to login and download the file):

http://www.vmware.com/support/developer/vcli/

Most recent at time of this post is: VMware-vSphere-CLI-5.0.0-615813.exe (https://my.vmware.com/group/vmware/get-download?downloadGroup=VCLI50U1)

2. Obtain the Dell OpenManage package for ESXi:

v4.0 OM-SrvAdmin-Dell-Web-6.3.0-2075.VIB-ESX40i_A00.9.zip
v4.1 OM-SrvAdmin-Dell-Web-6.5.0-2247.VIB-ESX41i_A01.zip
v5.0 OM-SrvAdmin-Dell-Web-7.1.0-5304.VIB-ESX50i_A00.zip


ESXi 4.0: http://ftp.dell.com/sysman/OM-SrvAdmin-Dell-Web-6.3.0-2075.VIB-ESX40i_A00.9.zip
ESXi 4.1: http://ftp.dell.com/sysman/OM-SrvAdmin-Dell-Web-6.5.0-2247.VIB-ESX41i_A01.zip
ESXi 5.0: http://www.dell.com/support/drivers/us/en/84/DriverDetails/DriverFileFormats/Product/poweredge-2950?DriverId=Y0WHR&FileId=2990390081&urlProductCode=False

3. Launch ‘vSphere CLI’ via Start > VMware > VMware vSphere CLI > Command Prompt.
4a. On ESXi 4.X, do the following:
- In the CLI window, type "cd bin" and hit enter to change the directory to the "bin" subfolder
- type "Vihostupdate.pl --server IpAddressOfHost -i -b PathToVIBinstallPackage" and hit enter.
(eg: Vihostupdate.pl --server 192.168.0.1 -i -b C:\OM-SrvAdmin-Dell-Web-7.1.0-5304.VIB-ESX50i_A00.zip)

4b. On ESXi 5.X, do the following:

- Copy the zip file containing the VIB to a datastore that the host can access. I used the vSphere client to connect to the host, then hit the configuration tab, Storage, then right clicked on the default local storage and hit "browse datastore". From that window I created a new folder named "updates" and uploaded the file into it.

- Take note of the Datastore location and name that is shown under "Datastore Details" in the bottom pane while the datastore you uploaded the file to is selected. My local store was /vmfs/volumes/502e86dd-dfa6c7f0-9ce2-0022197a5bef ...

- In the CLI window, type "esxcli --server IpAddressOfHost software vib install -d PathToVIBinstallPackage" and hit enter.
(eg: C:\Program Files (x86)\VMware\VMware vSphere CLI>esxcli --server 172.16.8.201 software vib install -d  /vmfs/volumes/502e86dd-dfa6c7f0-9ce2-0022197a5bef/updates/OM-SrvAdmin-Dell-Web-7.1.0-5304.VIB-ESX50i_A00.zip)


5. The system will prompt for the username ( root - then press enter) and the root password (then press enter). The update will now be applied to the ESXi server. The update will take approx 5mins on ESXi 4, and only moments on ESXi 5. (You can view the progress on the vSphere Client screen if you have it open).

Note, you may need to enable CIM providers as well... 4.1 and 5.0 do this automatically, but I think certain builds of 4.0 require you do it manually. You can do this by opening up the CLI again and typing CIMoemProviderEnabled = 1 (thanks to a comment below for pointing this out!)


6. Once completed, close the command prompt screen and reboot the ESXi server. Once the server has come back from being rebooted, using the vSphere Client, right click on the server which is in (maintenance mode) and choose “Exit Maintenance Mode”.


Section 3: Connect to the server via Server Administrator


1. Download the latest version of Dell Open Manage Server Administrator by entering the following into your web browser:

(Version 6.3) http://search.dell.com/results.aspx?s=biz&c=us&l=en&cs=555&k=managednode%2Cv.6.3.0&cat=sup

(Version 7.1) http://downloads.dell.com/FOLDER00574377M/1/OM-SrvAdmin-Dell-Web-WIN-7.1.0-5304_A00.exe

and searching for the right package for your needs.

For windows it is:

(6.3) OM-SrvAdmin-Dell-Web-WIN-6.3.0-2075_A00.20.exe (125MB)
(7.1) OM-SrvAdmin-Dell-Web-WIN-7.1.0-5304_A00 (186MB)

2. Install Server Administrator
3. Run Server Manager and choose "Manage a Remote Node", connect via IP Address, enter in the root and root password, and ensure "Ignore Certificate Warnings" is ticked.

NOTE: If your OMSA is responding extremely slow, as mine was, and you are using the Dell Customized version of ESXi 5.0 update 1, look at this article, which details an update to fix some CIM issues. You'll need to use vSphere Update Manager to install a patch on your hosts. http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2016538

Thursday, July 12, 2012

More on Exchange 2003 and 2010 Migration and Coexistence

A while back I made a post on migrating from SBS2003 to SBS2011, and the steps required to get coexistence working under that specific scenario, namely revolving around getting OWA redirection working. Those requirements definitely still apply here, but there were a few assumptions made based on some of the default setup work that SBS does for you. Here are some additional 'notes' about doing the same migration and coexistence without SBS factored in (So just a strait Exchange 2003 standard to Exchange 2010 standard migration.)

1) You need to configure your Exchange 2003 legacy URL. There isn't a GUI option for this... you have to issue the following command:

Set-OwaVirtualDirectory -identity "Exchange2010\owa (default web site)" -Exchange2003Url https://legacy.domain.com/exchange
Obviously, you should put your own server and domain name in there. This step is a no brainer... I just felt like documenting it here so that I don't have to look up the command syntax anymore. :P

2) This took me a while to figure out because the fix is completely non-intuitive. Upon configuring your Exchange 2003 legacy url and testing it out, you get an "HTTP 500 Internal Server Error" with no further information. The url stops at https://(legacyurl)/exchweb/bin/auth/owaauth.dll.

To fix this you must enable FBA (forms based authentication) on your Exchange 2003 frontend server before OWA redirection will work. I found the following video showing the steps required: http://www.youtube.com/watch?v=B8NAmFqGOl4

Basically, you need to follow all the steps in my previous post, and in addition open up the Exchange 2003 management console,  expand your way down to Administrative groups > first administrative group > servers > servername > Protocols > HTTP > Exchange Virtual Server, right click on the virtual server and go to properties. Under the settings tab, check the box for "Enable Forms Based Authentication" and click ok. After that, issue an IISRESET to restart your default website.

IMPORTANT: If you have only one Exchange 2003 backend server, and you configure SSL or FBA, you will likely break any activesync connections when you enable FBA. Here is an article explaining the situation from microsoft: http://support.microsoft.com/default.aspx?scid=kb;en-us;817379

Now... that seems like the proper way to go about things, but the major issue I noticed with my particular environment is that it was authentication that was broken for the mobile uses. A quick investigation of IIS after enabling FBA showed me that the 'windows authentication' option for the Exchange virtual directory was turned off. Checking the box fixed my problem! I think the only reason this worked for this particular site was because they did not have the "require SSL" box checked for the Exchange virtual directory. Not best practices, but this server will be gone soon anyway. As always, your mileage may vary. It's probably safer (or at least more supported) to follow the article I linked above.


Wednesday, May 9, 2012

SBS 2011 and SBS 2003 Exchange Coexistence Mode

This is another nitpicky little set of steps that I always forget about and end up googling to get a solution to... so I figured I would finally write it down!

During the migration from SBS 2003 to SBS 2011, there is likely some point where you will have a number of users still on 2003, and a number of users on your new 2011 box. Wouldn't it be great if everyone could get their email, access OWA, etc etc regardless of their mailbox location, thus lessening the pressure to get everything done in one 'big bang' migration? Sure it would. Well, here's the steps (although admittedly not very detailed... just high level... individual steps can be googled for more info):

1) Follow the SBS2003 to SBS2011 migration guide from microsoft up to the point where you are ready to Migrate your mailboxes. At this point, Microsoft guides you through a disruptive cutover rather than a smooth migration, so some tweaks are needed.

2) Make sure your SSL is set up appropriately on both servers. There are a few details that are paramount in getting this right. For one, you need to have an SSL certificate on both servers. Each server must also have a unique SSL certificate, as they cannot both resolve to the same name... otherwise this won't work right. If you want to migrate your existing SSL cert from SBS2003 to SBS2011 that's fine... just make sure you remove it from 2003 at the end of it, and generate a new one (either paid, which is a waste, or using the SBS2003 "Connect to the Internet" wizard to generate a self-signed one).

3) Make sure your DNS zones are set up appropriately so that each server is resolvable by their SSL certificate name. If you followed the SBS2011 wizard, it should have done this for you automatically by setting up an authoritative zone for servername.domain.com in your internal DNS server. You should do the same for your legacy SBS2003 server and it's new SSL certificate name.

4) Go into the properties of the Exchange virtual directory under your Default Website in IIS on your SBS 2003 box and check the Directory Security tab. Hit the Edit button under Authentication and Access Control and ensure that "Windows Authentication" is checked, in addition to whatever else was already there. If it's not checked... check it! This allows the two exchange servers to pass authentication to eachother... without it, your users will get prompted for authentication twice.

5) On your SBS 2011 server, open up your Exchange 2010 management shell and enter the following command: SetVirtualDirectory "Servername\owa (Default Web Site)" -Exchange2003Url https://OldServerName.Domain.com/Exchange . That tells your Exchange 2010 server where to send OWA requests when the user's mailbox is on the 2003 server.

Done! Now, just reconfigure your firewall to send the HTTPS/SMTP/Whatever requests to your new exchange server instead of your old one, and you should be off to the races. Resume following the SBS2011 migration guide, ignoring any steps you already did!