Progressive Neural Representation for Sequential Video Compilation

Neural Implicit Representations (NIR) have gained significant attention recently due to their ability to represent complex and high-dimensional data. Unlike explicit representations, which require storing and manipulating individual data points, implicit representations capture information through a learned mapping function without explicitly representing the data points themselves. They often prune or quantize neural networks after training to accelerate encoding/decoding speed, yet we find that conventional methods fail to transfer learned representations to new videos. This work studies the continuous expansion of implicit video representations as videos arrive sequentially over time, where the model can only access the videos from the current session. We propose a novel neural video representation, Progressive Neural Representation (PNR), that finds an adaptive substructure from the supernet for a given video based on Lottery Ticket Hypothesis. At each training session, our PNR transfers the learned knowledge of the previously obtained subnetworks to learn the representation of the current video while keeping the past subnetwork weights intact. Therefore it can almost perfectly preserve the decoding ability (i.e., catastrophic forgetting) of the NIR on previous videos. We demonstrate the effectiveness of our proposed PNR on the neural sequential video representation compilation on the novel UVG8/17 video sequence benchmarks.