{"id":2731,"date":"2023-12-19T12:18:22","date_gmt":"2023-12-19T17:18:22","guid":{"rendered":"https:\/\/blog.lufamily.ca\/kang\/?p=2731"},"modified":"2023-12-23T19:10:46","modified_gmt":"2023-12-24T00:10:46","slug":"ext4-fs-errors-on-nvme-ssd","status":"publish","type":"post","link":"https:\/\/blog.lufamily.ca\/kang\/2023\/12\/19\/ext4-fs-errors-on-nvme-ssd\/","title":{"rendered":"EXT4-fs Errors on NVME SSD"},"content":{"rendered":"\n

In my previous post<\/a>, I replaced my NVME boot disk on our media server thinking that the disk was defective because the file system (EXT4-fs) was reporting numerous htree_dirblock_to_tree:1080<\/code> errors.<\/p>\n\n\n\n

The errors continue to persist with the new disk, so I can eliminate the possibility of hardware as the cause of the issue.<\/p>\n\n\n\n

I noticed that the htree_dirblock_to_tree:1080<\/code> errors were caused by the tar<\/code> command and the time in which these errors occur coincided when the media server is being backed up. Apparently, the backup process is causing these errors with the tar<\/code> command.<\/p>\n\n\n\n

This backup process has remained unchanged for quite some time and has worked really well for us. I guess for some reason there is a bug in the kernel<\/code> or in the tar<\/code> command that is not quite compatible with NVME devices.<\/p>\n\n\n\n

I had to resort to finding an alternative backup methodology. I ended up using the rsync<\/code> method instead.<\/p>\n\n\n

\nsudo rsync --delete \\\n  --exclude 'dev' \\\n  --exclude 'proc' \\\n  --exclude 'sys' \\\n  --exclude 'tmp' \\\n  --exclude 'run' \\\n  --exclude 'mnt' \\\n  --exclude 'media' \\\n  --exclude 'cdrom' \\\n  --exclude 'lost+found' \\\n  --exclude 'home\/kang\/log' \\\n  -aAXv \/ \/mnt\/backup\n<\/pre><\/div>\n\n\n
It looks like this method is faster and can perform incremental backup. However, instead of backing up to an archive file, which I later need to extract and prepare during the restoration process, I have to back it up to a dedicated backup device. Since the old NVME disk is perfectly fine, I reused it as my backup device. I have partitioned this backup device in the same layout as the current boot disk.<\/p>\n\n\n\n
Device          Start        End    Sectors   Size Type\n\/dev\/sdi1        2048    2203647    2201600     1G Microsoft basic data\n\/dev\/sdi2     2203648 1921875967 1919672320 915.4G Linux filesystem\n\/dev\/sdi3  1921875968 1953523711   31647744  15.1G Linux swap<\/code><\/pre>\n\n\n\n
The only exception is that the first partition is not marked as boot<\/code> and esp<\/code>, so during the restoration process I will have to mark that partition accordingly with the parted<\/code> command by using the following commands:<\/p>\n\n\n\n
set 1 boot on\nset 1 esp on<\/code><\/pre>\n\n\n\n
The idea is that at 3am every night\/morning, I will backup the root filesystem to the second partition of the backup drive. If anything happens with the current boot disk, the backup drive can act as an immediately available replacement, after a grub-install<\/code> preparation as mentioned in the previous article<\/a>.<\/p>\n\n\n\n
Let us see how this new backup process works and hopefully, we can bid a final farewell to the htree_dirblock_to_tree:1080<\/code> errors!<\/p>\n\n\n\n
Update: 2023-12-22<\/h4>\n\n\n\n
It looks like even with the rsync<\/code> command, the htree_dirblock_to_tree:1080<\/code> errors still came back during the backup process. I decided to upgrade the kernel from vmlinuz-5.15.0-91-generic<\/code> to vmlinuz-6.2.0-39-generic<\/code>. Last night (2023-12-23 early morning) was the first backup after the kernel upgrade, and no errors were recorded. I hope this behavior persists and it is not a one-off.<\/p>\n","protected":false},"excerpt":{"rendered":"
In my previous post, I replaced my NVME boot disk on our media server thinking that the disk was defective because the file system (EXT4-fs) was reporting numerous htree_dirblock_to_tree:1080 errors. The errors continue to persist with the new disk, so I can eliminate the possibility of hardware as the cause of the issue. I noticed … Continue reading “EXT4-fs Errors on NVME SSD”<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[111],"tags":[97,5,28,6],"class_list":["post-2731","post","type-post","status-publish","format-standard","hentry","category-tech","tag-linux","tag-nas","tag-technology","tag-ubuntu"],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p7V6i8-I3","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/blog.lufamily.ca\/kang\/wp-json\/wp\/v2\/posts\/2731","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.lufamily.ca\/kang\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.lufamily.ca\/kang\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.lufamily.ca\/kang\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.lufamily.ca\/kang\/wp-json\/wp\/v2\/comments?post=2731"}],"version-history":[{"count":8,"href":"https:\/\/blog.lufamily.ca\/kang\/wp-json\/wp\/v2\/posts\/2731\/revisions"}],"predecessor-version":[{"id":2748,"href":"https:\/\/blog.lufamily.ca\/kang\/wp-json\/wp\/v2\/posts\/2731\/revisions\/2748"}],"wp:attachment":[{"href":"https:\/\/blog.lufamily.ca\/kang\/wp-json\/wp\/v2\/media?parent=2731"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.lufamily.ca\/kang\/wp-json\/wp\/v2\/categories?post=2731"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.lufamily.ca\/kang\/wp-json\/wp\/v2\/tags?post=2731"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}