Last modified: 2025-07-21 21:45:28
< 2025-07-20 2025-07-22 >I've made the steering wheel:
Great success, pleased with this. Didn't really have any mishaps, just designed it in CAD, cut it out with the CNC machine, bent it, bolted it together. Brilliant.
The bolts holding it together are just threaded into the plastic and they are quite short threads, and have partially stripped. I am thinking I'll reprint the plastic parts in black, and use heat-set inserts instead of threading into the plastic. This is fine for now though.
So the current status is I can't import the emails into SQLite because of NUL bytes in strings, https://github.com/rails/rails/issues/55373 has been fixed with raising an error on strings that contain NUL bytes.
So probably what I want to do is work out what is raw data and what is strings. For raw data make it have "blob" type, for strings sanitise them first.
Actually, the issue is that the email in question didn't have a text part, only had an application/zip attachment. So decoding that attachment and
treating it as "body text" is wrong. Fixing that makes it progress further. And beyond that, yeah, I want safe_utf8() to strip out NUL bytes.
And then instead of inserting all of the emails from all time, I want it to only go back a week or so. And I don't want to retrieve duplicates, so I probably need to track the IMAP server's UIDs. What do they look like and how do I look at them?
OK, doing that now. But I need to skip uids that I haven't seen before.
It is kind of lame that Rails "migrations" work the way they do. If I add a column, that is one migration, and if I then add an index on it, that is a second one. Even if I haven't committed my code yet, you can tell that I did it as 2 separate steps because it becomes 2 files. And it's not clear what the correct workflow would be to fix that.
OK, good, fine, I think my email fetcher is good enough for now. Next up is an initial inbox view.
We just want a page that lists all of the emails. Sweet, I have a page listing all of the emails, and I can click on one and it takes me to a page showing the content of that email.
Missing features:
We could kind of go either depth-first or breadth-first. Do I make it do all the things it needs to do, and then make it do them well? Or do I make the inbox exactly how I want it before I even start on a composer?
I guess go breadth-first. I just want to add a manual "Get new messages" button, and then work on a composer for sending emails.
I am actually really impressed with Ruby on Rails, it takes hardly any code to accomplish things.
So, a composer. For now we can have the act of sending an email just block until the email is sent. Obviously we'd eventually want sending and receiving emails to run as "background jobs", but with feedback on when they're complete.
Lol, I'm stuck at "Authentication failed" again!
Lol, I had the wrong password again. Wrong differently this time. Email sending is now working, woohoo!
So I can look at the inbox, refresh the inbox, and send emails. But the emails I send are not visible to me. I think I need the fetcher to fetch emails from all boxes, not just inbox.
What happens if we just don't call imap.select()? "No mailbox selected" error.
So maybe do we loop over all mailboxes? I'm doing that now, but it still doesn't seem to be noticing emails that I have sent. But it is still noticing new emails in the inbox. How is this possible?
Oh... maybe you can only fetch uids from the currently-selected mailbox. I'm picking out the uids I want from all mailboxes, and then asking for them all at once. But "Inbox" happens to be the last one I look at, so I only get the ones that exist inside the Inbox.
Great, working now.
I probably ought to do user isolation. So instead of grabbing the user out with User.first, we need to do... what?
I used https://github.com/heartcombo/devise - seems easy enough. And looks to be working, but I've just noticed that when I send an email it doesn't get written to the Sent folder. So I probably need to do that. I kind of need to decide whether I am trying to make a proper client that "plays nicely with others", or whether I am just using IMAP for convenience but I actually don't care what is in it. Probably it is nicer to play nicely.
So when I send an email I also need to copy it to the "Sent" mailbox. Currently I have EmailFetcher doing IMAP and EmailSender doing
SMTP. Do I want EmailSender to also operate IMAP and push up the new email? Or do I want to rename them both based on their protocols,
and have the IMAP module do it? Or a third module for EmailUploader? Stop overthinking it. Just stick it in EmailSender.
Wow, adding the email to the Sent box is only 4 lines of code. Ruby is so great.
I am upset about how slow fetching new emails is.
I think the issue is a lack of pipelining. I want to send the "select inbox, list uids 7 days, select sentbox, list uids 7 days" etc. all upfront, and then parse the responses when they come back, instead of 2 round-trips for every mailbox.
I could try launching a separate connection for each mailbox. Unsure if my mail server has a limit on concurrent connections.
Yes, it does:
Maximum number of connections from user+IP exceeded (mail_max_userip_connections=10)
OK, forget that then. I could bump the number but this is obviously not the way. Maybe forget about it for now.
Maybe I'll faff about with styles for a little bit and then stop for the day.
Cursor did a pretty decent job at a first pass of styling. Emma just sent me something to print and I couldn't do it from my tool because it doesn't show attachments, so let's try and do that.
How do attachments work? I have got Active Storage or something. The Email model has a field called "attachments", created with "has_many_attached". How do I look at them?
The rails console is really helpful. Email.attachments_attachments gives you a list of attachments. OK.
But this isn't exactly the same as Thunderbird's conception of attachments, because I have an email from an estate agent that my tool is saying has 6 attachments and Thunderbird is saying has none. I think maybe if an attachment is an image that is used inline in the email, Thunderbird maybe doesn't consider it an attachment? I don't think I immediately care.
How do I get the filename and size of the attachment? And a download link? I can't work it out from poking in the console. All I see is:
#<ActiveStorage::Attachment:0x00007492475bf3d0 id: 216, name: "attachments", record_type: "Email", record_id: 1234, blob_id: 216, created_at: "2025-07-21 15:17:11.505435000 +0000">
OK, attachments_attachments is some sort of implementation detail. You just want email.attachments.length, and then for each
one you want attachment.filename. It actually works if you just do the stupidest thing you might guess:
<% @email.attachments.each do |attachment| %>
<li><%= link_to attachment.filename, attachment %></li>
<% end %>
Great stuff. And attachment.byte_size to get the size.
We still need to be able to attach files to emails we send. That can come later.
Next up, let's try to render HTML emails. Obviously we won't load any external content. We can allow inline images though. Maybe inline images is a hassle because we have to fix up the URLs. So maybe skip inline images for now, let's just get styling working.
OK, easy enough with the sanitize() function - except this is loading external images, so not actually acceptable. It is also
showing comments on the screen where they should be hidden. And some emails have empty for both body_html and body_text,
even though they work in Thunderbird, unsure what is wrong there. Also line breaks look to be much too tall on the estate agent email.
So:
Also, if I allow use of the "style" attribute, does that potentially allow emails to draw things all over my page?
I've fixed the line spacing by setting margin-top/margin-bottom to 0 in the email display area.
OK! Nearly got attached images showing up, just the size is wrong. Fixed by allowing width/height in inline CSS.
Great success, this is working really well!
It would be cool to add a "share link" so you can share an email directly with somebody else, without forwarding it.
OK, better way to do HTML sanitisation: put it in an <iframe sandbox>. That stops it running JavaScript.
I still need to check all the CSS and image tags to make sure it's not loading external content.
Next things to look at: