Have you ever want to split a large file into several small files? I’ve face this problem few days ago. I need to split a large file (3GB log file) into several smaller file where i can read it using normal text editor.
To split large file into several smaller files, you can use split command in linux. Just follow the steps below and you will be able to split large file into smaller files.
-
in your shell key in
$ split --bytes=1m /path/to/large/file /path/to/output/file/prefix
- Done. You just split your large file into several smaller files
* You can change the output file size by changing the –bytes=1m to your preference. You can use b, k, or m. b represent bytes, k represent kilobytes, m represent megabytes.
To restore the original file, you can use cat command.
To join all the smaller file to restore the original file type:-
$ cat prefix* > NEWFILENAME
[tags]linux split file, split large file, file splitter, linux file splitter, split large file to smaller file, split large file linux[/tags]
Hi, I have a large file in Linux is 235821863 line,I want to remove the file line 4788139,I then convert the file into two smaller files,Each file contains a header that is,How do I use the command
Thanks for the info – it saved my day!
Hi All
I want to know how do i seprated out the list of file in linux
like in one dir. i have list of file *.log and some with *.txt and spme *.sh
how do i seprated out all any command sepcific option in ls cmd
please give an example
Thanks! One of the most useful commands I’ve found! 🙂
Your article is helpful, but slightly wrong. It should read :
split -–bytes=1m /path/to/large/file /path/to/output/file/prefix
an extra – in front of bytes..
Bipin Bahuguna: if it’s text file then probably u can compress it using tar gz. then split into smaller files.
i don’t have experience working with such a large file yet. so the solution might not be working, anyway it’s always good to try.
Good luck 🙂
Hi,
My application created up to 400gb log file.
is there any way to compress it .any suggestion?
Thanks,
Bipin Bahuguna
thanks guys, all the post was really useful..
Hi guys is there any way to split a file without knowing its size in specific pieces. For ex file.txt splitted into 4 pieces.
If not is any way to get the size (only the size) to divide it
thanks
zip -s 5000000 output input1 input2 etc.
I think froff meant the really single – .
From the split manpage:
“With no INPUT, or when INPUT is -, read standard input.”
For example, it tells split to read from the pipe (created by the ‘|’) instead of a specified file. Or it tells tar to write to standard output.
By the way, this article / discussion is really helpful!
the – is the way to include additional options.
it’s very common in linux and unix
hello
What does that ‘-‘ single character mean in command line?
regards
Whoops, guess I should have proof read that.
After the tar commands, and parameters (jcf, cf, cfz, etc.) and after the -, you’ll of course need to tell it what file(s) to use and/or what directory and after the split command and parameters, just before the output filename, be sure to use the final – as well.
Here’s a better, yet simple example (remove quotes):
“tar cvfp – filename.iso |bzip2 -9 -c |split -a 3 -db 500M – split_file.tbz.”
Attention to: Grahack
gzip:
tar cfz – |split -b 100M split_file.tgz. (note the last dot, optional though)
gzip with level 9 compression:
tar cf – |gzip -9 -c |split -b 100M split_file.tgz.
bzip2:
tar jcf |split -b 100M split_file.tbz.
bzip2 with level 9 compression:
tar cf – |bzip2 -9 -c |split -b 100M split_file.tbz.
then simply cat split_file* > file.tgz or .tbz and untar/gunzip or bunzip2 accordingly.
In most Linux distro’s, this should work as is. In OpenBSD or FreeBSD, I think the syntax may be a little different – the man pages will say though. As for other *nix OS’s, YMMV. =)
Hope this helps…
Found it:
http://www.linuxquestions.org/questions/linux-software-2/tar-split-question-605013/
Hi all,
resurrecting this old interesting post to ask a question about tar.
I’m creating an archive doing:
tar cfz /media/disk/0912171726.tar.gz /data
Is there a way to use ‘split’ and a pipe to create a few files WITHOUT creating the big file first?
for 3gp files they have a header that quicktime or other players look for to be able to play it. So you cannot use split to split it to 2 3gp files ever you change the extension. You have to use a special splitter for media files that copy also the header of the file.
not really familiar with 3gp file.
but i think u can try Ulead media studio, or Adobe Premier.
Both are good for video editing.
Okay, thanks for your reply.
Do you have any suggestions for a suitable video editor?
Jim
hi jim, once u split the file, you cant play it in any player.
You should use Video editor to cut the file and re-render it.
This linux command is not suitable for ur case.
Oh and one more thing is I’m using cygwin on winXP to perform the split.
Jim
How should I split a ‘.3gp’ file so that I can upload to facebook without exceeding their upload size limit?
I tried using split –bytes1024K V181109_13.50.3gp pref
.. but this produced two files both of which Quicktime can not play.
I renamed the files with a 3gp suffix before I tried playing them but that, I think is not relevent.
Jim
Thanks! I had to import a ~50 MB sql file on a remote server. It would take too long for the sql to script to run and throw security errors. It was difficult to work with a ~50MB sql file as most text editors would lock up. This command, with some slight alteration, worked great as I was able to split the file up into smaller chunks and import chunk by chunk… thanks for the tip!
What would the command be to split a 2.52gig file in half in Linux?
While you can’t use the Type command, you can use copy /b on windows:
copy /b “file.01″+”file.02″+”file.03” “file”
The /b switches the command into binary mode.
can’t use “type” on windows if you’re working with binary files. great for ascii though. type translates some control characters so output is not the same as input for binary files, which therefore end up corrupted.
[…] ref: http://www.techiecorner.com/107/how-to-split-large-file-into-several-smaller-files-linux/ […]
[…] split -b200m book.rar book-split Here 200m is 200 MegaByte book.rar is the source file to be splitted. and book-split is prefix of generated file. To restore the file: cat book-split* > NEWFILENAME Source: http://www.computerhope.com/unix/usplit.htm http://www.techiecorner.com/107/how-to-split-large-file-into-several-smaller-files-linux/ […]
ah forgot to say thanks i needed that to split a huge file i needed to give it to a friend on 2 512Mb USB-Flashs
the command i figured it out because it’s dos equivalent to cat
it’s better like that then to use zip multipart, it’s make u cooler in your friends eyes 😉 (using linux makes u smarter )
actualy in windows you can do:
$> type filepart1 filepart2 > filefull
I’m using cygwin on WXP to try to put my split files back together, but I cannot cat a file larger than 4GB. Is there another way?
[…] check this link. […]
If winxp does not have the copy command you can download the linux cat command for windows. Go to http://unxutils.sourceforge.net/ and download the unix utils, they’re free.
Then you can restore the files just as with the linux example above.
Very nice, but I need to recover the original file in a windows XP system. Command copy not found in execute prompt. What can I do then? Thx.
Just in case anybody needed (I did): if you use this command to split a binary file on *NIX and then copy the output to a DOS-aware system (Windows, for that matter) you can concatenate all the chunks using:
$> copy /B chunk* output
thanks for this post! i found it ’cause i needed it.
one thing, there should be two dashes before ‘bytes’.
split –bytes=2m /large/file /smaller/files/prefix