Lossy DNG isn’t so bad

Have you ever converted a RAW file to lossy DNG and noticed how much smaller it was? You can make the RAW file 80-90% smaller! I always thought of lossy DNG files as JPEG files saved at a compression level of 10 or 100%. In other words, with barely anything stripped, but it’s more interesting than that.

I discovered that a panorama I made in Lightroom was a lossy DNG. After I made it, I tried converting it to lossy DNG, and the new file was the same size as the old one. So, panoramas are lossy DNG. They used to be huge files, but, in Lightroom 13.0 that changed. Adobe started using Jpeg XL instead of Jpeg to store data in lossy DNG files. Even though the files are smaller, they are higher quality than older panoramas! It’s the same with HDR and Denoise. Remember when Denoise was introduced and the files it made were gigantic?

HDR images created by Lightroom are also now saved using the same lossy DNG format. An HDR image I created in 2017 on my 24MP Sony A7III is 77MB, but using Lightroom 13.3.1 that file is only 11.9MB. When I convert the image manually to lossy DNG with an embedded medium Jpeg and fastload data, the file reduces further to 8.1MB and looks practically the same at 100%.

I haven’t noticed any major problems fixing highlights or shadows in panoramas, so lossy DNGs are pretty good, but not without their own drawbacks, which I’ll get to later.

On the Adobe forums someone noticed this last year, and received a few interesting replies:

The compression method for derived DNGs has been changed from JPEG to JPEG-XL, which provides a smaller data footprint without loss of quality.

Creating a merged DNG such as a panorama is already a “lossy” process because the merged pixel data has been demosaiced, aligned, and blended from the original photos. Using JPEG XL compression makes a much smaller visual change.

Do not look at the words ‘lossy’ and ‘lossless’, look at your image and see if there is any visible effect of this new compression method. Recreate an older panorama and compare the old massive DNG with the new, much smaller DNG. Do you have any reason to be concerned?

Before you convert all your RAW files to lossy DNG, be aware that it will affect how Lightroom treats your file in ways you mightn’t think of. Years ago, I noticed that the Transform tool worked differently on lossy DNG files. It straightened walls slightly differently, not that it looked wrong, just different. You also can’t feed DNG files to Topaz Photo AI (Files->Plugins Extra->Process with Topaz Photo AI) or Lightroom Denoise, so use those before making your panorama or making the file a lossy DNG.

This is an example of a simple image that transforms differently when it was converted to a lossy DNG file. Lens correction has been applied to both images, and they are identical then. However, when “auto” transform is applied, the lossy DNG is modified differently. This bug has been there for years. If you want to compare yourself, here is the before image, and here’s the after image.

Greg Benz has published a post on the subject too and came up with the idea of exporting images to lossy DNG “to the same folder” and adding them back to the catalogue, making it easier to compare before and after images before you delete files. He points out some other problems I never noticed with lossy DNGs, such as haloing and colour tones, and other unspecified issues with older RAW images imported before 2012.

Is it worth converting your RAW files to lossy DNG? You’ll make savings of up to 90% and that’s very tempting. A 66MB RAW file may become a 9MB lossy DNG file. If you don’t like deleting, but you’re sure you’re not going to use an image, then it may be an option for you. On the other hand, use with caution on photos you want to edit and publish.

A comment on this video suggests loading the original photo and the lossy version into Photoshop as layers and use the “difference” blend more on the top layer. You’ll notice tiny changes between the images. Since I have deleted the original RAW file of the photo above, here’s one of the lighthouse at Ballycotton.

Here’s what the difference looks like in Photoshop.

There are obvious differences when you look at it like this, but I still think it’s worth considering. Today’s photo on my photoblog, The Rocks of Ballycotton, is a lossy DNG file for example.

And finally, one thing to consider is that third-party software support for lossy DNG files might not be what you expect. I haven’t tested any of the open-source software out there, so I’m not sure how good it is. A comment on that video I linked to above says RawTherapee handled them, and that was 6 years ago, so I’m sure the situation has improved since then.

While in the process of researching this, it has been so nice to find actual open, public, online forums and blogs talking about this, and not just on Reddit either. I bet there must be plenty of conversation about it hidden in Facebook groups too. Hidden from prying eyes.

Debug code left on the SEAI website

I filled in the solar PV grant form on the SEAI website this afternoon. I tried doing it this morning, but there was a “permission denied” error on their dashboard, which stopped things working. There had been a warning that they were doing updates overnight, so I figured it was probably teething problems with the update.

Hours later, upon submitting the grant form, I saw a “Paused on debugger statement” popup. Clicking play on it continued the process, and it all appears to work fine now. One wonders what happened to just submitting a form and processing everything on the server and refreshing the page?

webdev popup saying "Paused on debugger statement" with step and play buttons.
A chunk of Javascript code with a line saying "debugger" highlighted.

Anyone know someone on the SEAI dev team who can remove that debugger command? The joys of developing debugging on a live site.

Howto: Geotag photos in Lightroom with Google Timeline data

One feature in Adobe Lightroom that many don’t use often is the Map module. It’s really useful if your camera has GPS, if it can connect to your phone GPS, or if you manually drag images onto a map. I use the Sony “Imaging Edge Mobile” app to connect my camera to my phone via Bluetooth, but this drains the battery quickly when I’m out for a long time. It’s not ideal. If I want to use the Bluetooth remote control (not the app, the physical remote) I can’t use the location feature.

The changing of the guard. A guard approaches the camera. His dress uniform is a red cap with a long black silk tuft. His uniform is brown, and his shoes have a black tuft at the top. He has a grave expression on his face. Another guard stands in the background, facing away.
The changing of the guard at the Tomb of the Unknown Soldier, Athens, Greece.

The battery life of my camera when “Airplane mode” is enabled is stellar. Not so much when using a Bluetooth connection. If I know I’ll be out for an entire day, I’ll toggle it on and off and usually remember to do it on both devices, and sometimes I even make sure to check the next image has embedded GPS data!

Google Timeline is a feature that records your location data over time, using your phone’s GPS. This data can be very useful for geotagging photos because it provides a detailed record of where you’ve been. I’ve long wondered if I could use that GPS data to geotag my photos. You can export the entirety of your location data from Google using their Google Takeout tool. If you’ve had a Google account for a long time, that file will probably be well over 1GB is size if you had location sharing enabled. Unfortunately, it’s not in a format that Lightroom understands, and may contain many years worth of location data.

A few months ago, I decided to try writing a script that convert the Google Timeline JSON data into the GPX format that Lightroom wants, but I didn’t get very far, as “real life” intruded, and it was yet another project left on the long finger. I was looking forward to a day when I had sufficient time and energy to look at it.

The ruins of the Temple of Poseidon. A tree stands in the foreground, as well as some blocks with the prominent structure of columns and some horizontal pieces visible in the background.
The Temple of Poseidon in Sounion, Greece.

I’m glad I dropped it because I found someone had already created such a script, and it can consume a gigantic Records.json file. It will spit out the GPS data for any date range I want in GPX format. I discovered a Lightroom extension that makes importing that file easy, and it massages the data if a photo is taken in between points too.

The conversion script is a Python one called location-history-json-converter that can be found here. It’s relatively easy to install on Linux or macOS. If you use Windows you hopefully know how to get Python installed.

Once installed, I ran the following command to extract the GPS data I wanted for the first 17 days of May this year, and output it in the GPX format.

./location_history_json_converter.py Records.json records.gpx -f gpx -s 2024-05-01 -e 2024-05-17

That produced a nice 1.3MB file from the huge Records.json file.

The Maps module in Lightroom does allow you to import tracklogs, but I’ve used it in the past, and sometimes it missed photos, maybe because my tracklog wasn’t recording points often enough. This time I used Jeffrey Friedl’s geoencoding extension for Lightroom. The ability to adjust the timezone was critical to me, as I wanted to geoencode or geotag the photos I took on a recent trip to Greece.

A train covered in graffiti with people visible through the windows.
On the metro in Athens, Greece.

I messed up the first time and told it to adjust the timezone by 2 hours, the difference between Ireland and Greece but that was an hour out, possibly because of daylight savings time settings somewhere in the mix of camera, phone and exported data. I was really annoyed with myself over that because the plugin allows you to choose only “unmapped photos”, but now every photo was mapped. 75% of them were mapped to the wrong location!

I realised it didn’t matter because the originally geotagged images were tagged off my phone, so they had the same location data as Google Timeline. I went slower a second time, picking a couple of test images of places I knew, and by adjusting the timezone by 3 hours, I saw that it worked perfectly. I ran the job again against more than 1800 images and those images were geotagged correctly.

In the past, I used a GPS logger app to log my location to my phone. That created a GPX file, but I completely forgot I had that installed and simply used the Sony app connected to my camera for a long time. The Google Timeline data is already there because I allow Google to log that data, so it’s a handy source of this information. I can attempt to geotag years worth of photos now, but I’ll have to go slow, as I know the time on my camera drifts, so that will have to be accounted for.

We’re out of water

That’s not really an announcement you want to hear from the pilot of your flight, but that’s what happened on my flight from Athens to Amsterdam recently.

A photo from the window of a KLM flight. The wing is clearly visible and the KLM logo is etched on the end. The plane is flying above the clouds with a clear blue sky above and a hazy horizon.

The pilot came on the intercom and informed his passengers there was no water to flush the toilets or wash hands. He said that’s the first time he’s had to make such an announcement. The flight had been delayed an hour at Athens, and then we spent 25 minutes on the ground while the congested air over Europe cleared. So I’m surprised someone forgot to fill the water tank.

Water bottles were provided, but I think everyone was glad to get to Amsterdam, even if several people missed their connection due to delays. Thankfully, the flight to Cork was late in the evening, and I had time to make it to gate D6. The airport was mostly closed down with only a few shops open, and passport control was backwards which was odd but worked well enough to get us to our destination. The police there were friendly and efficient.

I have to say, one thing I love about KLM is their free WiFi that allows messaging on WhatsApp, Messenger and other apps. It was great to message my family and friends while flying. It’s something that will make me choose KLM for a connection if all other things are equal. The flight to Cork didn’t have it, but the outgoing flight a week earlier did.

I was watching Top Gun: Maverick on my phone, downloaded on the Netflix client, and it looked like my flight would land before the film ended.

Screenshot from Top Gun Maverick showing the text "Hang on. We gotta get low." Tom Cruise is flying a fighter jet while the world turns around him.

It was a late flight, the lights switched off in the cabin. I was staring at the action on my phone, with headphones hiding the drone of the aircraft. I was glued to the action on my phone, totally absorbed in it. Maverick shouts at his passenger, “Hang on. We gotta get low.” during a dogfight. A moment later, I feel my flight touching down on the runway at Cork Airport, and I look up in mild panic. Talk about immersion. Thanks, KLM! Perfect timing!

Watch out for Aurora Tonight

Map of Europe showing the strength of the Aurora likely visible.

If you live north of me in Europe, there’s a pretty good chance you’ll see an aurora tonight. I’m looking forward to seeing all the photos!

The US Space Weather Prediction Center has issued its first “severe geomagnetic storm watch” today for the first time in nearly 20 years.

The activity is expected to make the aurora borealis phenomenon visible across Northern Europe in the early hours of Saturday, May 11th (after midnight tonight).

Donegal Daily

Someone already posted fabulous photos from Latvia!

Aurora in the skies over a house in Latvia
Aurora photo by Simron.

Writing C64 ASM in VS Code

Thanks to this Hackaday article, I found a video describing how to use Visual Studio Code to write 6502 assembler for the Commodore 64!

It’s been a long time since I’ve done any ASM on the C64, and compared to the programming environments available these days, I was working in the dark ages with my Action Replay monitor, typing code directly into memory.

Kick Assembler is used to assemble the code into a .prg file, using the KickAss (C64) extension for VS Code to launch the emulator with the code loaded into memory. I see Kick Assembler 8-Bit Retro Studio is mentioned in the comments, and it is indeed still being developed, so it might be the better choice, but either will make developing on the C64 much simpler.

It gives a nice overview of using Git to track changes too, something I would have also loved on the C64. So many files prefixed with numbers as I experimented with effects …

Faulty Towers – The Dining Experience

Sybil and Basil Fawlty
BASIL!
Yes, dear!

If you’re a fan of Fawlty Towers, the hit British comedy from the 70s, then you’ll love The Dining Experience. Apart from the entertaining “shows” the three actors put on, you’ll, of course, be eating a three-course meal.

Manuel, the hapless Spanish waiter, doing his best to do his job.
I know nothing!

In Cork, they performed in the Metropole Hotel, so the food was great. Portions were small but it’s the experience you’re paying for and I have no hestitation in recommending it if you’re at all familiar with the show!

Speculating on what to load next

The Speculative Loading plugin for WordPress is a plugin you should probably try out on your site, especially if you use WP Super Cache or Jetpack Boost to cache things. It uses the new speculation API that Chrome/Edge supports to load pages in the background if you even hover over a link.

It will dynamically prefetch or prerender pages before they’re requested by the visitor on your site, which means that the page will show instantly when the visitor actually clicks the link.

It doesn’t work in Firefox yet, but it won’t hurt either, as the browser will just ignore the extra bits and pieces added to the page.

The default “moderate” eagerness works fine for me. The “eager” setting appeared to load links if the cursor got anywhere near them, which was a little too aggressive.

You won’t notice your browser loading the page in the Network tab of the webdev tools, but if you tail your access_log, you’ll see the requests go through when you hover over the links.

Browse around this site, or take a look at my photoblog for a feel of what it does.

There’s more info in the make blog post about it, and this insightful comment about the wastefulness of loading pages that might not be used, especially for visitors on limited data plans, or low powered devices. That’s definitely something to think about before using this plugin. I may yet remove it later, and I’ll update this post if I do.

Close

Ad-blocker not detected

Consider installing a browser extension that blocks ads and other malicious scripts in your browser to protect your privacy and security. Here are a few options.

  • uBlock Origin is a free, open source, ad blocker for your browser.
  • Use pi-hole if you have a spare Raspberry Pi on your network.
  • Set the private DNS settings on your phone to dns.adguard.com to block adverts and trackers.