In this paper, we consider a generalized extensible bin packing problem with overload cost, first proposed by Denton et al. in 2010, in which the total size of items packed into a bin is allowed to exceed its capacity, and the cost incurred each bin is equal to the fixed cost plus the overload cost, the objective is to minimize the total cost of all bins. According to the characteristics of the problem, we first propose an improved ant colony optimization algorithm (IACO), which enhances the positive feedback effect of ACO by improving the update method of pheromone and the adaptive adjustment parameters. We also introduce a variable neighborhood search method in ACO to improve the convergence of the algorithm and get rid of the phenomenon of local extrema. Then, we present a discrete particle swarm optimization algorithm (DPSO) to solve the problem. In order to ensure the uniform distribution and high quality of the initial particle swarm, we use some heuristic methods in the initialization process of the swarm, so that the initial particle can cover the entire search space with a large probability, which effectively improves the performance of DPSO algorithm. Finally, we compare and analyze the performance of these proposed algorithms through two sets of computational experimental frameworks. Compared with some algorithms in the literature, computational results signify that the improved ACO algorithm and MDPSO algorithm are more competitive than some other metaheuristic algorithms.