Python List All Files in a Directory

In this tutorial, we have compiled a variety of Python methods to list all files in a directory such as os.walk(), os.listdir(), os.scandir(), glob(), and a recursive function. Each method includes a self-explanatory example so that you can grasp them easily. Towards the end, you’ll find a table comparing these methods and advising which is the most suitable.

You may need such techniques, especially in Selenium Python automation or working with configuration/log files. Python comes with the default OS module that enables several functions to interact with the file system. As mentioned above, it has a walk() method which lists all files inside a directory in Python. Besides, it has another function listdir() that finds files on the specified path.

Similarly, Python’s Glob module has a glob() method that checks for the specified files in the current directory. Let’s now have a deeper look at each of the methods for listing all files in a directory.

Also Check: Loop Through Files in a Directory using Python

Python Methods to List All Files in a Directory

Here, we are demonstrating functions that help traverse the file system and search for the files present.

Os.walk() method

It gathers the file names present in a directory by traversing the dir in either top-down or bottom-up. It returns a tuple of the following three:

Root: Gets only the folders from the input.
Dirs: Gets sub-directories from the root.
Files: Gets all files from the given root and directories.

Find all text files in dirs and subdirs

Below is the sample Python code listing all files in given directories and sub-directories.

import os

location = 'c:/test/temp/'
files_in_dir = []

# r=>root, d=>directories, f=>files
for r, d, f in os.walk(location):
   for item in f:
      if '.txt' in item:
         files_in_dir.append(os.path.join(r, item))

for item in files_in_dir:
   print("file in dir: ", item)

After execution, the following is the result:

c:/test/temp/notes/readme.txt
c:/test/temp/release/artifact_list.txt
c:/test/temp/dist/doc/readme.txt
c:/test/temp/dist/samples/sample.txt

List all dirs under given dirs and subdirs

Check the below Python code to find and print all dirs under the given dir/subdir.

import os

location = 'c:/test/temp/'

dirs_in_dir = []

# r=>root, d=>directories, f=>files
for r, d, f in os.walk(location):
   for item in d:
      if '.txt' in item:
         dirs_in_dir.append(os.path.join(r, item))

for item in dirs_in_dir:
   print("Dirs under dir: ", item)

After execution, the following is the result:

c:/test/temp/notes/
c:/test/temp/release/
c:/test/temp/dist/
c:/test/temp/dist/doc/
c:/test/temp/dist/samples/

Glob.glob() method

Many times, we have to iterate over a list of files in a directory having names matching a pattern. In such a case, the Python glob module helps capture the list of files in a given directory with a particular extension.

glob() function

This function fetches a list of files filtered based on the given pattern in the pathname. We can take a pathname which is absolute as well as relative. The wild cards such as * and ? are also allowed symbols.

Another parameter, recursive is off (false) by default. If its value is True, then this function searches inside all subdirectories of the current directory and finds files having the desired pattern

List all files in the current directory having “.py” extension

For example – The following Python code lists all files in the current directory having the “.py” extension.

import glob

location = 'c:/test/temp/'

fileset = [file for file in glob.glob(location + "**/*.py", recursive=True)]

for file in fileset:
    print(file)

After execution, the following is the result:

c:/test/temp/notes/get_sample.py
c:/test/temp/release/test1.py
c:/test/temp/dist/doc/core.py
c:/test/temp/dist/samples/first_sample.py

Read about Python glob in more detail.

Get all dirs in a specified dir and subdirs

import glob

location = 'c:/test/temp/'

folderset = [folder for folder in glob.glob(location + "**/", recursive=True)]

for folder in folderset:
    print(folder)

After running the above code, the following is the result:

c:/test/temp/notes/
c:/test/temp/release/
c:/test/temp/dist/
c:/test/temp/dist/doc/
c:/test/temp/dist/samples/

Os.listdir() method to list text files

It gives a list including the names of the files in the directory specified in the location (path). The list happens to be in random order. It excludes the ‘.’ and ‘..’ if they are available in the input folder.

import os

location = 'c:/test/temp/'

for file in os.listdir(location):
    if file.endswith(".txt"):
        print(os.path.join(location, file))

After execution, the following is the result:

c:/test/temp/notes/readme.txt
c:/test/temp/release/artifact_list.txt
c:/test/temp/dist/doc/readme.txt
c:/test/temp/dist/samples/sample.txt

To learn Python in a step-by-step manner, read this Python tutorial.

Using Path.iterdir()

The function Path.iterdir() is present in Pathlib module. It is a newer module in Python 3.4 that provides a more robust way to work with paths. The Path.iterdir() method can be used to list the files and directories in a directory.

Let’s find out how this Python module works to list the files in a directory.

from pathlib import Path

# Get a list of all the files in the current directory
files = Path().iterdir()

# Print the list of files
for file in files:
    print(file)

Using the os.scandir() function

The os.scandir() is also a relatively new function that returns a generator of directory entries. This can be useful if you need to iterate over the files in a directory in a more efficient way.

Go through the below Python code which demonstrates how to use os.scandir() to list all files in a directory.

import os

def list_files_with_details(directory):
    """Lists all of the files in a directory and prints some additional details about each file, such as the file size and the file type."""

    # Get a generator of directory entries
    entries = os.scandir(directory)

    # Iterate over the directory entries and print some details about each file
    for entry in entries:

        # Get the file size
        file_size = entry.stat().st_size

        # Get the file type
        file_type = entry.stat().st_mode

        # Print the file name, file size, and file type
        print(f"{entry.name}: {file_size} bytes ({file_type})")

# List the files in the current directory and print some additional details about each file
list_files_with_details(".")

This code example will print a list of all of the files in the current directory, along with the file size and file type for each file. You can modify this code example to print any additional information about the files in the directory that you need. For example, you could print the file creation date, the file’s last modified date, or the file permissions.

Also Check: Read File Line by Line in Python

Python recursive method to list all files in a directory

In Python, to list the files in a directory and all of its subdirectories, we can use a recursive function. The following function will list all of the files in a directory and all of its subdirectories, recursively:

import os

def list_files_recursively(directory):
    """Lists all of the files in a directory and all of its subdirectories, recursively."""

    files = []

    # Iterate over the files and directories in the directory
    for entry in os.scandir(directory):

        # If the entry is a directory, recursively list the files in that directory
        if entry.is_dir():
            files += list_files_recursively(entry.path)

        # If the entry is a file, add it to the list of files
        elif entry.is_file():
            files.append(entry.path)

    return files

# Get a list of all the files in the current directory and all of its subdirectories
files = list_files_recursively(".")

# Print the list of files
for file in files:
    print(file)

Comparing different methods

Here is a table to compare all methods, their performance, how one can decide which one to choose, and which is best:

The performance of these methods will depend on the number of files in the directory and the complexity of the search pattern. In general, the os.scandir() function will be the most efficient to iterate over the files in a directory. It does this by avoiding unnecessary system calls and by caching file information.

The performance of the glob module will depend on the complexity of the search pattern. If the search pattern is simple, then it will be relatively fast.

Conclusion

There are many ways to list files in a directory in Python. The best method to use will depend on your specific needs. If you just need to list the files in a directory, the os.listdir() function is the simplest option. However, if you need to list the files in a directory and all of its subdirectories, you can use the os.walk() method. When you need to list the files in a directory based on a pattern, you can use the glob module. If you need to iterate over the files in a directory in a more efficient way, you can use the os.scandir() function.