Although photovoltaic (PV) energy production offers several environmental and commercial advantages, the irregular nature of PV energy can challenge the design and development of the energy management systems. Precise forecasting for PV energy production is therefore of vital importance to supply consumers to improve trust in functionality of the energy management system. Stimulated by current developments in deep learning (DL) techniques as well as the promising efficiency in energy-related applications, this study introduces a novel DL architecture, called PV-Net, for short-term forecasting of day-ahead PV energy. In PV-Net, the gates of the gated recurrent unit (GRU) are redesigned using convolutional layers (called Conv-GRU) to enable efficient extraction of positional and temporal characteristics in the PV power sequences. The Conv-GRU cells are stacked in bidirectional (Bi-dir) blocks to enable modeling temporal information in forward and backward directions. The Bi-dir block is residually connected to avoid information loss across layers and to facilitate gradient flow during training. A real-world case study from Alice Springs, Australia, is employed to evaluate and compare the performance of the proposed PV-Net against recent cutting-edge approaches. The values of four performance measures demonstrate the efficiency of the proposed PV-Net in terms of prediction precision and consistency.